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Design of experiments 

DOF 

Degree-of- freedom 

DR 

Discrepancy report 

EF 

External failure 

ETA 

Event tree analysis 

EVOP 

Evolutionary operation 

FMEA 

Failure modes and effects analysis 

FMECA 

Failure modes, effects, and criticality analysis 

FT A 

Fault tree analysis 

IF 

Internal failure 

Uy) 

Foss function (quality) 

FCF 

Fower control limits 

FDF 

Fower decision line 

FIB 

Farger is better 
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LSL 

Lower specification limit 

MTBF 

Mean time between failures 

MTBR 

Mean time between repairs 

MTTR 

Mean time to repair 

NASA 

National Aeronautics and Space Administration 

NGT 

Nominal group technique 

NIB 

Nominal is best 

PDA 

Probabilistic design analysis 

PHA 

Preliminary hazard analysis 

PRA 

Probabilistic risk assessment 

PRACA 

Problem reporting and corrective action 

OSHA 

Occupational Safety and Health Administration 

QFD 

Quality function deployment 

RBD 

Reliability block diagram 

RSM 

Response surface methodology 

SE 

Standard error 

SESTC 

System Effectiveness and Safety Technical Committee 

SIB 

Smaller is better 

SME 

Sum of mean error 

SMQ 

Safety and mission quality 

SMR 

Sum of mean replicate 

SPC 

Statistical process control 

SRM 

Solid rocket motor 
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SSE 

Sum of squares error 

SSR 

Sum of squares replication 

SST 

Total sum of squares 

STA 

Success tree analysis 

TQM 

Total quality management 

UCL 

Upper control limit 

UCLR 

Upper control limit range 

UDL 

Upper decision line 

USL 

Upper specification limit 

WFA 

Work flow analysis 
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REFERENCE PUBLICATION 


SYSTEM ENGINEERING “TOOLBOX” FOR DESIGN-ORIENTED ENGINEERS 


1. INTRODUCTION 


1.1 Purpose 

Many references are available on systems engineering from the project management perspective. 
Too often, these references are of only limited utility from the designer’s standpoint. A practicing, 
design-oriented systems engineer has difficulty finding any ready reference as to what tools and 
methodologies are available. 

The purpose of this system engineering toolbox is to provide tools and methodologies available 
to the design-oriented systems engineer. A tool, as used herein, is defined as a set of procedures to 
accomplish a specific function. A methodology is defined as a collection of tools, rules, and postulates to 
accomplish a purpose. A thorough literature search was performed to identify the prevalent tools and 
methodologies. For each concept addressed in the toolbox, the following information is provided: (1) 
description, (2) application, (3) procedures, (4) example, if practical, (5) advantages, (6) limitations, and 
(7) bibliography and/or references. 

This toolbox is intended solely as guidance for potential tools and methodologies, rather than 
direction or instruction for specific technique selection or utilization. It is left to the user to determine 
which technique(s), at which level of detail are applicable, and what might be the expected “value 
added” for their purposes. Caution should be exercised in the use of these tools and methodologies. Use 
of the techniques for the sake of “using techniques” is rarely resource-effective. In addition, while 
techniques have been categorized for recommended areas of use, this is not intended to be restrictive. 
Readers are encouraged to question, comment (app. A) and, in general, use this reference as one source 
among many. The reader is also cautioned to validate results from a given tool to ensure accuracy and 
applicability to the problem at hand. 


1.2 Scope 

The tools and methodologies available to the design-oriented systems engineer can be 
categorized in various ways depending upon the application. Concept development tools, section 2, are 
useful when selecting the preferred option of several alternatives. Among these alternatives are such 
things as cost, complexity, weight, safety, manufacturability, or perhaps determining the ratio of 
expected future benefits to the expected future costs. 

System safety and reliability tools, section 3, address the following areas of concern: (1) identify 
and assess hazards, (2) identify failure modes and show their consequences or effects, and (3) symbolic 
logic modeling tools used to understand the failure mechanisms of the system. These tools are also used 
to determine the probability of failure occurring or the reliability that a component will operate success- 
fully, either in comparative or absolute terms, as applicable. 

Design-related analytical tools, section 4, are applied to show (1) which parameters affect a sys- 
tem the most or least, (2) a method for specifying dimensions and tolerances, and (3) the determination 
of the possibility or probability of having form, fit, or function problems with a design, or to determine a 
tolerance or dimension necessary to avoid these problems. 
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When there is a desire to monitor performance, identify relationships, or reveal the most 
important variables in a set of data, graphical data interpretation tools are typically applied. These tools 
are discussed in section 5. Statistical tools and methodologies, section 6, compare sample statistics and 
population statistics. Variations are identified and mathematical relationships are determined. Many 
excellent texts are available on statistical methods, as are software packages. For this reason, this 
document touches only lightly on this area. 

Total quality management (TQM) tools, section 7, are applied to continuously improve perfor- 
mance at all levels of operation, in all areas of an organization, using all available human and capital 
resources. Finally, quantitative tools that are used to identify potentially hazardous conditions based on 
past empirical data are trend analysis tools, section 8. The ultimate objective for these tools is to assess 
the current status, and to forecast future events. 

To assist in further defining optimal areas in which each technique may be useful, table 1-1 
provides a functional matrix which categorizes the functionality of each tool or methodology into (1) 
data analysis, (2) problem identification, (3) decision making, (4) modeling, (5) prevention, (6) creative, 
and (7) graphical. These functionality categories are found in reference 1.1. 

Extensive research was performed in order to identify all prevalent tools and methodologies 
available to the design-oriented systems engineer. Nevertheless, important tools or methodologies may 
have been overlooked. If a tool or methodology should be considered for this toolbox, appendix A is 
provided for the reader to complete and return to the individual indicated on the form. 

To further illustrate how selected tools and methodologies in this toolbox are applied, and misap- 
plied, appendix B provides a case study illustrating the trials and tribulations of an engineer applying his 
recently acquired knowledge of the techniques to a given work assignment. 

Appendix C provides a glossary of terms applicable to the tools and methodologies in this toolbox. 


1.3 Relationship With Program or Project Phases 

Each tool or methodology may be performed in a minimum of one of the following phases, as 
described in reference 1.2, of a project design cycle. 

(1) Phase A (conceptual trade studies) — a quantitative and/or qualitative comparison of 
candidate concepts against key evaluation criteria to determine the best alternative. 

(2) Phase B (concept definition) — the establishment of system design requirements as well as 
conceptually designing a mission, conducting feasibility studies and design trade-off 
studies. 

(3) Phase C (design and development) — the initiation of product development and the 
establishment of system specifications. 

(4) Phase D (fabrication, integration, test, and evaluation) — system verification. 

(5) Phase E (operations) — the deployment of the product and performance validation. 

Table 1-2 provides a project phase matrix for all of the tools and methodologies identified in this 
toolbox. An entry of (1) for the phase means the technique is primarily performed in that phase and an 
entry of (2) means the technique is secondarily performed in that phase. Though the entries in this matrix 
are a result of research by the authors, other phases should be considered by the user for a particular tool 
or methodology. 
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Table 1-1. System engineering “toolbox” function matrix — Continued 
















































































Table 1-1. System engineering “toolbox” function matrix — Continued. 



8.2 Problem trend analysis 

8.3 Programmatic trend analysis 

8.4 Supportability trend analysis 

8.5 Reliability trend analysis 

Note: Functionality categories found in reference 1.1. 
















































































Table 1-2. System engineering “toolbox” project phase matrix — Continued 



fote: Phases discussed in reference 




























































































Table 1-2. System engineering “toolbox” project phase matrix — Continued 



Note: Phases discussed in reference 1.2. 





























































































Table 1-2. System engineering “toolbox” project phase matrix — Continued. 



8.4 Supportability trend analysis 

8.5 Reliability trend analysis 

Note: Phases discussed in reference 1.2. 
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2. CONCEPT DEVELOPMENT TOOLS 


Trade studies and cost-versus-benefit studies are presented in this section. These tools are used to 
select the preferred option of several alternatives. Trade studies, section 2.1, are quantitative and/or 
qualitative comparison techniques to choose an alternative when considering such items as cost, 
complexity, weight, safety, manufacturability, etc. Cost-versus-benefit studies, section 2.2, provide a 
method to assess alternatives by determining the ratio of expected future benefits to expected future 
costs. 


A summary of the advantages and limitations of each tool or methodology discussed in this 
section is presented in table 2-1. 


2.1 TRADE STUDIES 


2.1.1 Description 

In general, trade (or trade-off) studies provide a mechanism for systematic depiction of both sys- 
tem requirements and system design options for achieving those requirements. Once tabulated, a 
comparison of relevant data (cost, complexity, weight, safety, manufacturability, etc.) is then performed 
to rank those candidate design options in order of desirability. 

These studies are categorized as either a weighted factor trade study or an analytical hierarchy 
trade study, with the latter being a special version of the former. These techniques are described in 
reference 2.1. A trade tree can be generated with either of the above two options. A trade tree is simply a 
pictorial representation of how high-level alternatives (or issues) in the decision process are logically 
resolved into decreasingly lower level alternatives (or issues). A trade tree may be presented without 
results or simply as a representation of options. 

A weighted factor trade study is usually performed when each of the options under consideration 
is very well defined and there is good definition of the program requirements as well. All factors 
(program requirements) that are determined to be important, are delineated with an associated weighting 
factor. The options are then assessed with respect to each of the factors and an equation is developed that 
weighs this assessment. The decision is then based upon the numerical results of the analysis. 

The analytical hierarchy process (AHP) is a variation of the weighted factors analysis and is the 
most complex of the trade studies presented here. This approach allows for delineation of the facts and 
rationale that go into the subjective assessment of each of the options. Further, pseudoquantitative equa- 
tions may be developed (as in probabilistic assessment equations for failure causes in fault tree analyses) 
to increase confidence in analysis results. The AHP provides a multicriteria analysis methodology that 
employs a pairwise comparison process to compare options to factors in a relative manner. This is used 
when subjective verbal expressions (equal, moderate, strong, very strong, etc.) are easier to develop than 
numerical (3 versus 3.2, etc.) assessments. Pseudoquantitative numbers are then ascribed to the words 
and a score developed for each of the options. 

A key to any trade study is the initial selection and prioritization of specific desirable attributes. 
This is often very difficult and the prioritization delineation may change during the early phases of the 
program. It is very important, and often overlooked, that when the prioritization changes, a cursory look 
at the significant, completed trades should be performed to determine any impacts to their conclusions. 
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Table 2-1. Concept development tools and methodologies. 
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2.1.2 Application 


These studies should typically be performed in phase A of NASA projects. However, trade 
studies can also be performed in phase B, or whenever a method is needed to select alternatives, such as 
selecting test methods, evaluating design change proposals, or performing make-or-buy decisions. A 
trade study analysis allows a systematic approach to evaluation of design options with respect to 
programmatic considerations or other, nonreliability related considerations (weight, maintainability, 
manufacturability). These studies may also be used to help the designer delineate which system require- 
ments are most important (used in conjunction with the Pareto chart analysis, sec. 5.6). 

2.1.3 Procedures 


The procedures for performing a weighted trade study are presented below. By performing step 
6, an AHP weighted trade study will be performed. These procedures are described in detail and were 
adapted from reference 2.1. 

(1) Define the mission objectives and requirements for the system under consideration. These 
objectives and requirements should be clear, accurate, and specific. These requirements will 
provide the scope of the assessment and the basis for the selection criteria. Prioritize the 
objectives/requirements if possible; this will aid in the weight factors for the selection criteria. 

(2) Identify credible alternative candidates for the system under consideration. These 
alternatives can be imposed or obtained in brainstorming sessions (sec. 7.7). The list of 
alternatives selected during brainstorming sessions may be reduced by eliminating 
alternatives which do not appear capable of meeting requirements. The list may be reduced 
further by eliminating alternatives with low probability of successful implementation or 
those which are expected to exceed cost constraints. The remaining alternatives should be 
described in sufficient detail that the relative merits between them can be ascertained. 

(3) Develop a trade tree (optional). A trade tree is developed to graphically illustrate the 
alternatives and how high-level alternatives in the decision process are logically resolved 
into decreasingly lower level alternatives. For large trade studies with many alternatives 
and criteria attributes, create a trade tree to group alternatives with unique criteria 
attributes. A large trade study may be resolved into several smaller trade studies with fewer 
required total comparison evaluations. This will lead to fewer resources to conduct the 
assessment without degradation of the results. 

(4) Develop and specify the selection criteria to be used in the analysis. The selection criteria 
are benchmarks to assess the effectiveness and applicability characteristics of the 
alternatives to be considered. Ideally, the selection criteria should have the following 
characteristics: 

a. Be expressed in general terms that mean the same thing to every evaluator. 

b. Be practical to measure or predict within acceptable uncertainty and cost limits. 

c. Provide a distinction between alternatives without prejudice. 

d. Correlate directly to the established requirements and high priority issues. (A 
numbering system, showing the specific correlation, is often useful here.) 

e. Be separate and independent from each of the other selection criterion in all aspects of 
the assessment. 
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(5) Establish weights for the selection criteria. These weights should reflect the importance of 
each criterion relative to its importance to the overall selection decision. The weights 
should be given numerical values to accommodate objective comparisons between 
unrelated criteria. The numerical values of the weight factors should sum to 100. The 
weights should be predetermined by the person (or group) with the ultimate decision 
authority, but not necessarily shared with the analysts to ensure that alternatives are 
assessed against each criterion objectively. Each criterion may be resolved into several 
levels of components to establish its weight. The degree to which the individual criterion is 
resolved into components is dependent on how effective the criterion components can be 
evaluated, and represents the resolution limit of the assessment. 

Consult with the end user of the system (the internal or external customer) to verify that the 
selection criteria and weights are compatible with his needs. 

(6) Perform an analytical hierarchy process as described in reference 2.2 to establish weights 
for the selection criteria (optional). This technique is beneficial for very complex trade 
studies when operational data are not available and a subjective analysis is to be performed. 
The following steps define this process: 

a. Establish a scale of the relative level of significance to the system objectives between 
two given criteria attributes. Establish three to five definitions to subjectively define 
this scale of relative level of significance. Generate clarifications for each definition so 
that qualified managers and engineers can subjectively use the definitions. If five 
definitions are used, assign the numerical values 1, 3, 5, 7, and 9 to these definitions in 
order of increasing diversity between the given two attributes. Reserve the numerical 
values of 2, 4, 6, and 8 as values to be assigned when interpolating between two of the 
definitions. If attribute n has a numerical value of relative level of significance of 
relative to attribute m, then attribute m has a numerical value of relative level of 
significance of “1 //’ relative to attribute n. 

b. Survey a group of qualified managers and engineers (or customers) to establish a 
consensus on the relative relationships between each attribute and the rest of the 
attributes. 

c. Create a normalized matrix (all the attributes versus all the attributes) with these 
relationships. Note that all elements of the diagonal of this matrix equal 1. 

d. Determine the relative weights for each criterion component by performing an 
eigenvector analysis. 

e. Determine the weight for all attributes by calculating the product of each individual 
attribute weighing factor and its weights of associated category headings. 

(7) Generate utility functions (optional). This technique is used to establish a consistent scale 
for dissimilar criteria. A relationship is established between a measure of effectiveness for 
each selection criterion and a common scale (for example, 0 to 10). The relationship may 
be a continuous function (not necessarily a straight line) or discrete values. For attributes 
other than technical, such as cost, schedule, risk, etc., a subjective verbal scale may be used 
(i.e., high, medium, low). 

(8) Assess each alternative relative to the selection criteria. First estimate the performance of 
every alternative for a given criterion in terms of the measure of effectiveness used in gen- 
erating the utility functions. For the ideal situation, the analyst may use test data, vendor pro- 
vided data, similarity comparison, modeling, engineering experience, parametric 


2-4 



analysis, or other cost-effective and reliable methods to generate the performance estimates. 
In reality, this is often very difficult to perform objectively. It is worthwhile, however, even 
when somewhat subjective (i.e., heavy use of engineering experience). If quantification of 
qualitative ranking is required, use caution in drawing conclusions. Assume that a difference 
in the conclusion of less than one-half the quantified number of a one-step difference is an 
equivalent answer. This corresponds to a confidence band for the evaluation. 

Next, determine the score for each alternative relative to a given criterion by correlating the 
estimate of performance for all the criteria to the mutual scale using the utility functions 
generated in step 7. Next, multiply the scores for all alternatives by the weight factor for the 
criterion (determined in steps 5 or 6) to determine the weighted score for all alternatives for 
that criterion. Repeat this procedure for all criteria attributes. 

(9) Tabulate the results. Generate a matrix of criteria versus alternatives to summarize the 
results from the preceding steps. A typical table is illustrated in table 2-2 and was 
generalized from an example presented in reference 2.1. 


Table 2-2. Typical weighted trade study summary table. 2 1 


Criteria 

Alternates, x, (x, through x„) 

Criterion, 

yj 

Weights 

Y.Wj — 100 

Alternate x l 

Alternate x 2 

Alternate x 3 
Through 
Alternate x n _ , 

Alternate x n 
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(0-10) 

Weighted 

Score 

Score 

(0-10) 

Weighted 

Score 

Score 

(0-10) 

Weighted 

Score 

Score 

(0-10) 

Weighted 

Score 

>’i 
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S 21 
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S n2 

W 2 S n2 

A 
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S 23 

W 2 S 2 3 



S «3 

W 3 S n3 

>4 t0 > m _| 










y m 

W m 

^ lm 

^ ni* \m 

s 2m 

W 2 S 2 m 



C 

nm 

^ rn^nm 

Total 


UWjSij) 


Uw jSij )j 




UWjSij 


(10) Perform a sensitivity analysis to evaluate the merit of the results relative to making an alter- 
nate selection. Examine the results of the weighted trade study to see if any total weighted 
scores of any alternatives are closer in numerical value than is warranted in making a deci- 
sion due to the confidence levels of the performance estimates that had been used to estab- 
lished the scores. If this is the case, then gather more data to increase the confidence level of 
the performance estimates, repeat the assessment, and regenerate the summary table for the 
weighted trade study. If, after the analysis is repeated, the alternative numerical total score is 
too close to make a decision, reconsider the selection criterion and weighting factors. 

(11) Select the superior alternative. Select the alternative with the highest value of total 
weighted scores. 
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2.1.4 Example 
Problem: 


Four alternatives for a new automobile design are being considered. The selection decision will 
be based on comparing the four alternatives to the following criteria attributes and their associated 
weight factors: 


Item 

Criteria Attribute 

Weight Factor 

1 

Average fuel economy 

20 

2 

Acceleration (0 to 60 mph) 

15 

3 

Braking (70 to 0 mph) 

15 

4 

Road handling 

15 

5 

Implement new technology 
risk 

10 

6 

Cost 

_25 


Total 

100 


Utility functions have been generated for each criteria attribute and are presented in figure 2-1. 
The estimates for each alternative relative to each criteria attribute are listed below: 


Item 

Criteria attribute 

Measure of 
Effectiveness 

Alt. 

A 

Alt. 

_B 

Alt. 

_C 

Alt. 

_D 

1 

Average fuel economy 

miles per gallon 

16 

19 

23 

18 

2 

Acceleration (0 to 60 mph) 

seconds 

7 

9 

10 

12 

3 

Braking (70 to 0 mph) 

feet 

180 

177 

190 

197 

4 

Road handling 
(300 ft dia. skidpad) 

g 

0.86 

0.88 

0.83 

0.78 

5 

Implementing new 
technology risks 

— 

Fow 

Avg. 

High 

Very 

low 

6 

Cost 

Dollars, x 1,000 

21 

20 

24 

22 


From the information given above, formulate a weighted factor trade study summary table, and 
select the superior alternative. 


2-6 



Score 


Score 



10 


5 - 


0 

Ver y Avg. Very 

Low High 

Implementing New Technology Risks 


Score 


Score 



Acceleration (0 to 60 mph), seconds 



Score Score 




Figure 2-1. Example utility functions. 
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Solution: 


Presented in figure 2-2 is the completed weighted factor trade study summary. Scores were 
determined from effectiveness measures for all alternatives relative to all criteria attributes and the utility 
functions. Based on the results of the trade study, alternative B is the preferred option. 


Criteria 

Alternates, x t 

Attribute 

Item 

Weights 
Xwj =100 

Alternate A 

Alternate B 

Alternate C 

Alternate D 



Score 

(0-10) 

Weighted 

Score 

Score 

(0-10) 

Weighted 

Score 

Score 

(0-10) 

Weighted 

Score 

Score 

(0-10) 

Weighted 

Score 

1 

20 

1 

20 

4 

80 

8 

160 

3 

60 

2 

15 

9 

135 

8 

120 

7.5 

112.5 

5 

75 

3 

15 

9.8 

147 

9.9 

148.5 

8.5 

127.5 

5 

75 

4 

15 

4.5 

67.5 

7 

105 

2.5 

37.5 

1.5 

22.5 

5 

10 

8 

80 

6 

60 

4 

40 

10 

100 

6 

25 

4 

100 

5 

125 

1 

25 

3 

75 

Total 


549.5 


638.5 


502.5 


407.5 


Figure 2-2. Example weighted factor trade study summary table. 


2.1.5 Advantages 

The following advantages can be realized from performing trade studies: 

(1) Different kinds and/or levels of study allow flexibility in the depth of the review, i.e., 
resources expended can be commensurate with the benefits of the task. 

(2) This technique is adaptive to prioritization based upon programmatic considerations (cost, 
schedule) as well as technical ones (weight, reliability, etc.). 

(3) Identification of disadvantages of a specific design option may lead to the definition of 
effective countermeasures if combined with other techniques. 

(4) The method provides a clearly documented analysis in which the (a) prioritized objectives 
and requirements, (b) considered alternatives, and (c) selection methodology are recorded. 
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2.1.6 Limitations 


The following limitations are associated with performing trade studies: 

(1) These techniques are very dependent upon the expertise of the analyst and the amount of 
available accurate quantitative data. 21 

(2) Improper generation of selection criteria, weight factors, and utility functions can prejudice 
the assessment and lead to incorrect results. 2 1 

(3) The number of alternatives which can be considered is limited by the expenditure of 
resources to perform the analysis. 2 1 

(4) Options evaluated are not determined as a result of the study but must be decided upon 
prior to the assessment by the person (or group) with decision authority. 

(5) Weighting factors and advantages/disadvantages are very subjective (although objective 
data may be added in the analytical hierarchy process approach (AHPA), this significantly 
complicates and enlarges the study) and this subjectivism significantly influences the study 
conclusions. 


2.1.7 Bibliography 

Blanchard, B.S., and Fabreycky, W.J.: “System Engineering and Analysis.” Second edition, Englewood 
Cliffs, Prentice Hall, New Jersey, pp. 67-72, 1990. 

Cross, N.: “Engineering Design Methods.” John Wiley & Sons, pp. 101-121, 1989. 

Saate, T. L.: “Analytical Hierarchy Process.” McGraw-Hill, 1980. 


2.2 COST-VERSUS-BENEFIT STUDIES 


2.2.1 Description 

Cost-versus-benefit studies are also known as benefit-cost analyses, 2 32 4 benefit-cost ratio 
analyses, 2 5 and cost-benefit analyses. 2 6 Cost-versus-benefit studies, as described in reference 2.5, pro- 
vide a method to assess alternates by determining the ratio of expected future benefits to expected future 
costs. Both the expected future benefits and costs are expressed in terms of present value. The 
alternatives are ranked in decreasing order with the preferred option being the alternative with the 
highest benefit-to-cost ( B/C ) ratio, while falling within overall cost restraints. 


2.2.2 Application 

Benefit-cost analyses apply to the selection of projects 2 - 32 4 or machines or systems 2 52 6 based on 
their relative B/C ratios. Cost-versus-benefit studies, as discussed in this section, will apply to the 
selection of system or system element alternatives based on their relative B/C ratios. These studies 
should typically be performed in phase A, however, they could also be performed in phases B or C. 
These studies can be used when two or more alternatives are being considered with fixed cost 
constraints, fixed desired results or benefits, or when both costs and desired results vary. 
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2.2.3 Procedures 


The following procedures to perform cost-versus-benefit studies were adapted from references 
2.3, 2.5, and 2.6. 

(1) Define the requirements for the system or system element under consideration. These 
requirements should be measurable and verifiable. Translate general and vague 
requirements into specific, quantitative requirements in which system effectiveness can be 
measured and assessed. 2 6 Prioritize these requirements, if possible. 

(2) Define a list of credible, mutually exclusive alternatives; that is, if one alternative is selected, 
the others are not to be implemented. 2 - 3 Each alternative should be characterized to a level of 
completeness such that all substantial costs and benefits can be identified. 2 6 Note that the 
alternatives require an implicit determination of technical and schedule viability. 

(3) Develop and specify the selection criteria to be used in the analysis. The example selection 
criteria presented in table 2-3 were adapted from reference 2.5. 

Table 2-3. Example selection criteria for cost-versus-benefit analyses. 


Condition or Circumstance 

Selection Criteria 

Budget C is fixed 

Maximum B/C ratio. 

Desired result B is fixed. 

Maximum B/C ratio. 

Two alternatives are being considered with 
neither budget C or desired result B fixed. 

Calculate the AB-to-AC ratio between the 
alternatives. Choose the lower cost 

alternative, unless the AB-to-AC ratio is 
>1. Then choose the higher cost 
alternative. 

More than two alternatives are being 
considered with neither budget C or desired 
result B fixed. 

Select alternative per benefit-cost ratio 
incremental analysis (sec. 2.2.3, 
step 11). 


(4) Identify the cost or savings for each alternative. The cost should include such items as initial 
investment, and ongoing operating and maintenance expenses (including depreciation) for the 
life of the system. The savings should include such items as residual or salvage values, etc. 2 3 

(5) Identify the benefit or detriments for each alternative. The benefits might include such 
items as increased performance, reduced operating times, compressed schedules, increased 
reliability, increased safety, value added due to increase productivity, etc. The detriments 
might include such items as loss of production time, increased schedules, increased 
equipment operating costs, environmental impacts, reduced property value, etc. 2 - 3 The cost 
risk and technical maturity for each alternative may be included as a multiplying factor if) 
for this analysis. Since it is subjective, use of only three factors — 0.5, 0.75, or 1 — is 
probably as fine a distinction as is warranted. 

(6) Specify the time interval (expected operating life of the system) to which the analysis is to 
apply. 
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(7) Develop cost and savings estimates and the benefits and detriments estimates for each 
alternative. 2 3 The estimates for each alternative should be for the same time interval 
specified in step 6. Every attempt should be made to base cost and savings estimates on 
actual historical cost data. 

(8) Identify the interest rate that will be assumed for the analysis. 2 3 Convert all costs, savings, 
benefits, and detriments estimates to present worth values. 2 - 5 

(9) Determine the total cost for each alternative by algebraically summing all costs as positive 
values and all savings as negative values. 

(10) Determine the total benefit value for each alternative by algebraically summing all benefits 
as positive values and all detriments as negative values. 

(11) Calculate the B/C ratio for each alternative by dividing the total benefit ( B ) by the total cost 
(C). 

For cases with fixed cost restraints or fixed desired results or benefits, perform step 12. 2 5 

(12) Rank the alternatives relative to their respective cost-to-benefit ratios, and select the 
superior alternative based on selection criteria established in step 3. 

For cases with cost restraints and desired results or benefits that vary, perform steps 13 

through 17. 25 

(13) If there exists any alternatives with a B/C >1, then do not give further consideration to 
alternatives with a B/C <1. 

(14) Order the remaining alternatives in sequence of increasing total C. 

(15) Determine the incremental B/C ratio A B/C for each consecutive pair of alternatives with 
increasing total cost. 

A B/C i = ABj /A C, , where AB t = B, +1 - B i and AC,- = C,- +1 - C, 

for each z th pair of (n - 1) pairs of n alternatives where alternative i = 1,2 listed in order 
of C. 

(16) Next, examine each distinct increment of increased cost investment. If the A/I/C,- is <1, then 
the increment is not beneficial. If the A 5/C,- is >1, then the increment is beneficial. 

(17) The preferred alternative is the last alternate listed in order of increasing cost whose 
incremental A/I/C,- is >1. Therefore, the preferred alternative may not necessarily have the 
greatest BIC ratio. 


2.2.4 Example 
Problem: 

Five data acquisition systems (DAS) are under consideration to acquire data for solid rocket 
motor tests in a test stand over a 10-yr time interval. Each system has a different total cost and the 
capabilities of each system are different in terms of maximum number of channels, maximum sample 
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rates, required maintenance, data accuracy, turnaround time between tests, and mean time between 
system failures. The present value of the estimated total cost and total value of combined benefits of the 
system are presented below. The present values of cost and benefits were determined over a 10-yr 
expected system life, with an assumed annual interest rate of 10 percent. Perform a cost-versus-benefit 
analysis to determine the best alternative. 


System 

A 

B 

C 

D 

E 

Total cost (dollars) 

500k 

300k 

750k 

800k 

400k 

Total benefits (dollars) 

750k 

400k 

900k 

750k 

600k 

B/C 

1.50 

1.33 

1.20 

0.93 

1.50 


Solution: 

Step 1. Delete options with a B/C ratio <1. Since the B/C for system D is <1, this option will no 
longer be considered. 

Step 2. List the remaining options in order of increasing total cost. 


System 

B 

E 

A 

C 

Total cost (dollars) 

300k 

400k 

500k 

750k 

Total benefits (dollars) 

400k 

600k 

750k 

900k 


Step 3. Determine the incremental B/C ratio A B/C for each consecutive pair of alternatives with 
increasing total cost. 


Increment 

E-B 

A-E 

C-A 

A Total cost (dollars) 

100k 

100k 

250k 

A Total benefits (dollars) 

200k 

150k 

150k 

A B/C 

2.0 

1.5 

0.6 


Step 4. Identify the preferred alternative as the last alternate listed in order of increasing cost 
whose incremental A/I/C, is >1. 

By inspection, the last incremental A B/C with a value >1 is A-E. Therefore, the preferred 
alternative is DAS A. 
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2.2.5 Advantages 


The following advantages are realized by performing cost-versus-benefit analyses: 

(1) The analyst can assess the cost effectiveness of several alternatives over the entire life cycle 
of the proposed system under consideration. 

(2) The method provides a clearly documented analysis in which the prioritized 
objectives/requirements, the alternatives considered, and the selection methodology are 
recorded. 


2.2.6 Limitations 


Cost-versus-benefit analyses possess the following limitations: 

(1) The analysis is flawed if system requirements are incomplete or inadequate. If the system 
operating environment is not understood or accurately characterized, the total costs can be 
underestimated. If the system requirements are too general or vague, benefits cannot be 
addressed in specific, measurable terms of effectiveness. 2 5 

(2) The analysis is only as good as the list of alternatives considered. An incomplete list of 
alternatives will lead to an incomplete analysis. 2 6 

(3) The analysis is flawed if incomplete or inaccurate cost estimates are used. 2 6 

(4) The analyst must be able to quantify the value of benefits, which are often intangible or 
insubstantial and difficult to characterize in terms of monetary value. 2 3 

(5) The analysis does not take into account technical complexity or maturity of an alternative, 
except as a cost uncertainty factor. Further, system reliability and safety issues are not 
treated except by the selection of the alternative. As cost is generally only one of many 
factors, this tool is generally insufficient for selection of large, new design efforts, but more 
appropriate to production-level design solutions. 


2.2.7 Bibliography 

Thuesen, G.J., and Fabrycky, W.J.: “Engineering Economy.” Seventh edition, Englewood Cliffs, 
Prentice Hall, New Jersey, 1989. 
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3. SYSTEM SAFETY AND RELIABILITY TOOLS 


This section describes several system safety and reliability tools available to the system engineer 
analyst. The risk assessment matrix is discussed in section 3.1. This device supports a standard 
methodology to subjectively evaluate hazards as to their risks. It is used in conjunction with hazard 
analyses, such as the preliminary hazard analysis (PHA) technique discussed in section 3.2. The PHA 
can be used to identify hazards and to guide development of countermeasures to mitigate the risk posed 
by these hazards. The energy/flow barrier analysis discussed in section 3.3 is also a technique to identify 
hazards and to evaluate their corresponding countermeasures. 

Once hazards are identified, they can be further explored if failure modes of the elements of the 
system are known. The failure modes and effects analysis (FMEA), discussed in section 3.4, can be used 
to identify failure modes and their consequences or effects. Also discussed in section 3.4 is the failure 
modes, effects, and criticality analysis (FMECA). The FMECA is similar to the FMEA but also 
addresses the criticality, or risk, associated with each failure mode. 

Several symbolic logic methods are presented in this section. These methods construct 
conceptual models of failure or success mechanisms within a system. These tools are also used to 
determine either the probability of failures occurring or the probability that a system or component will 
operate successfully. The probability of a successful operation is the reliability. If the probability of 
failure ( P F ) is examined, then the model is generated in the failure domain and if the probability of 
success (P s ) is examined, then the model is generated in the success domain. For convenience, the 
analyst can model either in the failure or success domain (or both domains), then convert the final 
probabilities to the desired domain using the following expression: P F + P s = 1. 

These models are developed using forward (bottom- up) or backwards (top-down) logic. When 
using forward logic the analyst builds the model by repeatedly asking, “What happens when a given fail- 
ure occurs?” The analyst views the system from a “bottom-up” perspective. This means he starts by 
looking at the lowest level elements in the system and their functions. Classically, the FMEA, for exam- 
ple, is a bottom-up technique. When using backwards logic to build a model, the analyst repeatedly asks, 
“What will cause a given failure to occur?” The analyst views the system from a “top-down” 
perspective. This means he starts by looking at a high level system failure and proceeds down into the 
system to trace failure paths. The symbolic logic techniques discussed in this section and their 
characteristics are presented in table 3-1. 

Each of the symbolic logic techniques has its own unique advantages and disadvantages. 
Sometimes it is beneficial to construct a model using one technique, then transform that model into the 
domain of another technique to exploit the advantages of both techniques. Fault trees are generated in 
the failure domain, reliability diagrams are generated in the success domain, and event trees are 
generated both in the success and failure domains. Methods are presented in section 3.9 to transform any 
one of the above models into the other two by translating equivalent logic from the success to failure or 
failure to success domains. 

Probabilities are propagated through the logic models to determine the probability that a system 
will fail or the probability the system will operate successfully, i.e., the reliability. Probability data may 
be derived from available empirical data or found in handbooks. If quantitative data are not available, 
then subjective probability estimates may be used as described in section 3.12. Caution must be 
exercised when quoting reliability numbers. Use of confidence bands is important. Often the value is in a 
comparison of numbers that allows effective resource allocation, rather than “exact” determination of 
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Table 3-1. Symbolic logic techniques. 


Technique 

Section 

Success 

Domain 

Failure 

Domain 

Forward 

(Bottom-Up) 

Backwards 

(Top-Down) 

Reliability block 
diagram 

3.5 

V 



V 

Fault tree analysis 

3.6 


V 


V 

Success tree analysis 

3.7 

V 



V 

Event tree analysis 

3.8 

V 

V 

V 


Cause-consequence 

analysis 

3.10 

V 

V 

V 

V 

Directed graph matrix 
analysis 

3.11 

V 

V 


V 


expected reliability levels. Probabilistic design analysis (PDA) is discuss in section 3.14. This technique 
uses advanced statistical methods to determine Pp modes. 

Failure mode information propagation modeling is discussed in section 3.13. This technique 
allows the analyst to determine what information is needed, and how and where the information should 
be measured in a system to detect the onset of a failure mode that could damage the system. 

Finally, probabilistic risk assessment (PRA) is discussed in section 3.15. This is a general 
methodology that shows how most of the techniques mentioned above can be used in conjunction to 
assess risk with severity and probability. 

A summary of the major advantages and limitations of each tool or methodology discussed in 
this section is presented in table 3-2. 


3.1 Risk Assessment Matrix 


3.1.1 Description 

The risk assessment matrix, as described in reference 3.1, is a tool to conduct subjective risk 
assessments for use in hazard analysis. The definition of risk and the principle of the iso-risk contour are 
the basis for this technique. 

Risk for a given hazard can be expressed in terms of an expectation of loss, the combined 
severity and probability of loss, or the long-term rate of loss. Risk is the product of severity and 
probability (loss events per unit time or activity). Note: the probability component of risk must be 
attached to an exposure time interval. 

The severity and probability dimensions of risk define a risk plane. As shown in figure 3-1, iso- 
risk contours depict constant risk within the plane. 
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avoids accepting unknown, intolerable, and senseless skilled analysts. Techniques can be misapplied and 
risk. results misinterpreted. 

































SEVERITY 

and 

PROBABILITY, 

the 

two variables 
that 

constitute risk, 
define a 
RISK PLANE. 



Figure 3-1. Risk plane. 


The concept of the iso-risk contour is useful to provide guides, convention, and acceptance limits 
for risk assessments (fig. 3-2). 

Risk should be evaluated for worst credible case, not worst conceivable case, conditions. Failure 
to assume credible (even if conceivable is substituted) may result in an optimistic analysis; it will result 
in a nonviable analysis. 


3.1.2 Application 

The risk assessment matrix is typically performed in phase C but may also be performed in phase 
A. This technique is used as a predetermined guide or criteria to evaluate identified hazards as to their 
risks. These risks are expressed in terms of severity and probability. Use of this tool allows an organiza- 
tion to institute and standardize the approach to perform hazard analyses. The PHA, defined in section 
3.2, is such an analysis. 
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RISK ASSESSMENT 
CONVENTION: If 
possible, assess Risk for 
the Worst- Credible 
Severity of outcome. (It’ll 
fall at the top end of its 
own iso-risk contour.) 


© 


RISK ASSESSMENT 
GUIDES: If Risk for a 
given Hazard can be 
assessed at any severity 
level, an iso-risk contour 
gives its probability at all 
severity levels. (Most, 
but not all hazards 
behave this way. Be wary 
of exceptions — usually 
high-energy cases.) 


ACCEPTANCE: Risk 
Tolerance Boundaries 
follow iso-risk contours. 



PROBABILITY 


Figure 3-2. Iso-risk contour usage. 


3.1.3 Procedures 


Procedures, as described in reference 3.1, for developing a risk assessment matrix are presented 

below: 


(1) Categorize and scale the subjective probability levels for all targets, such as frequent, 
probable, occasional, remote, improbable, and impossible (adapted from MIL-STD- 
882C). 3 - 2 Note: A target is defined as the “what” which is at risk. One typical breakout of 
targets is personnel, equipment, downtime, product loss, and environmental effects. 

(2) Categorize and scale the subjective severity levels for each target, such as catastrophic, 
critical, marginal, and negligible. 

(3) Create a matrix of consequence severity versus the probability of the mishap. Approximate 
the continuous, iso-risk contour functions in the risk plane with matrix cells (fig. 3-3). 
These matrix cells fix the limits of risk tolerance zones. Note that not the analyst but man- 
agement establishes and approves the risk tolerance boundaries. 

(4) The following hints will be of help when creating the matrix: 

a. Increase adjacent probability steps by orders of magnitude. The lowest step, 
“impossible,” is an exception (fig. 3-4(a)). 

b. Avoid creating too many matrix cells. Since the assessment is subjective, too many 
steps add confusion with no additional resolution (fig. 3-4(b)). 
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Matrix cells approximate the 
continuous, iso-risk contour 
functions in the Risk Plane. 
Steps in the Matrix define Risk 
Tolerance Boundaries. 


A 


N 




J 


s 1 

E 

V 

E II 

R 

I 

T 

Y HI 


IV 


“Zoning” the Risk Plane into 
judgmentally tractable cells 
produces a Matrix. 




D 


PROBABILITY 


Figure 3-3. Risk plane to risk matrix transformation. 


c. Avoid discontinuities in establishing the risk zones, i.e., make sure every one-step path 
does not pass through more than one zone (fig. 3-4(c)). 

d. Establish only a few risk zones. There should only be as many zones, as there are 
desired categories of resolution to risk issues, i.e., (1) unacceptable, (2) accepted by 
waiver, and (3) routinely accepted (fig. 3-4 (d)). 

(5) Calibrate the risk matrix by selecting a cell and attaching a practical hazard scenario to it. 
The scenario should be familiar to potential analysts or characterize a tolerable perceivable 
threat. Assign its risk to the highest level severity cell just inside the acceptable risk zone. 
This calibration point should be used as a benchmark to aid in evaluating other, less 
familiar risks. 
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(a) Useful conventions. 


s 1 

F 

© 

E 

© 
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© Q 

B 

© 

A 

© 

V 

E II 
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© 

0 

© 

© 

© 

I 

T 
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© 

© © 

© 

© 

IV 

Q 


FLAWED | 

© 

© 


PROBABILITY 


Three zones will usually suffice. 

A Hazard’s Risk is either... 

• (3) Routinely Accepted 

• (2) Accepted by Waiver, or 

• (1) Avoided. 


A 24-cell Matrix can be resolved 
into 9 levels of “priority,” or even 
more. But what are the rational 
functions for the many levels? 



(b) Do not create too many cells. 

Figure 3-4. Helpful hints in creating a risk assessment matrix — Continued 
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PROBABILITY 


(c) Avoid discontinuities. 



PROBABILITY 


(d) Do not create too many zones. 

Figure 3-4. Helpful hints in creating a risk assessment matrix — Continued. 
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3.1.4 Example 


A typical risk assessment matrix, adapted from MIL-STD-882C, 3 2 is presented in figure 3-5. Example 
interpretations of the severity and probability steps for this matrix are presented in figure 3-6. 


Severity 

of 

Consequences 

Probability of Mishap** 

F 

IMPOSSIBLE 

E 

IMPROBABLE 

D 

REMOTE 

C 

OCCASIONAL 

B 

PROBABLE 

A 

FREQUENT 

i 

CATASTROPHIC 





© 


II 

CRITICAL 




© 



III 

MARGINAL 



© 




IV 

NEGLIGIBLE 








Risk Code/ Actions 

Imperative to suppress risk to lower level. 


© 




® 


Operation requires written, time-limited waiver, endorsed 
by management. 


Operation permissible. 


NOTE Personnel must not be exposed to hazards in Risk Zones 1 and 2. 


'Adapted from MIL-STD-882C "Life Cycle = 25 yrs. 

Figure 3-5. Typical risk assessment matrix. 
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3.1.5 


Advantages 


The risk assessment matrix provides the following advantages: 3 1 

(1) The risk matrix provides a useful guide for prudent engineering. 

(2) The risk matrix provides a standard tool of treating the relationship between severity and 
probability in assessing risk for a given hazard. 

(3) Assessing risk subjectively avoids unknowingly accepting intolerable and senseless risk, 
allows operating decisions to be made, and improves resource distribution for mitigation of 
loss resources. 


3.1.6 Limitations 

The risk assessment matrix possesses the following limitations: 3 1 

(1) The risk assessment matrix can only be used if hazards are already identified. This tool 
does not assist the analyst in identifying hazards. 

(2) This method is subjective without data and is a comparative analysis only. 

3.1.7 Bibliography 

Code of Federal Regulations, Medical devices, “Pre-Market Notification.” sec. 807.90, 
vol. 21. 

Code of Federal Regulations, “Process Safety Management of Highly Hazardous Chemicals.” sec. 
1910.119 (e), vol. 29. 

Department of Defense Instruction, No. 5000.36. “System Safety Engineering & Management.” 
NASA NHB 1700.1, vol. 3, “System Safety.” 

NUREG/GR-0005. “Risk-Based Inspection - Development of Guidelines.” 


3.2 Preliminary Hazard Analysis 


3.2.1 Description 

A PHA, as described in reference 3.3, produces a line item tabular inventory of nontrivial system 
hazards, and an assessment of their remaining risk after countermeasures have been imposed. This 
inventory includes qualitative, not quantitative, assessments of risks. Also, often included is a tabular 
listing of countermeasures with a qualitative delineation of their predicted effectiveness. A PHA is an 
early or initial system safety study of system hazards. 
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3.2.2 Application 

PHA’s are best applied in phase C but may also be applied in phase B. This tool is applied to cover 
whole-system and interface hazards for all mission phases. A PHA may be carried out, however, at any 
point in the life cycle of a system. This tool allows early definition of the countermeasure type and 
incorporation of design countermeasures as appropriate. 


3.2.3 Procedures 

A flowchart describing the process to perform a PHA is presented in figure 3-7. Procedures for 
performing PHA’s, as described in reference 3.3, are presented below: 

(1) Identify resources of value, such as personnel, facilities, equipment, productivity, mission 
or test objectives, environment, etc. to be protected. These resources are targets. 

(2) Identify and observe the levels of acceptable risk that have been predetermined and 
approved by management. These limits may be the risk matrix boundaries defined in a risk 
assessment matrix (sec. 3.1). 

(3) Define the extent of the system to be assessed. Define the physical boundaries and 
operating phases (such as shakedown, activation, standard operation, emergency shutdown, 
mainten-ance, deactivation, etc.). State other assumptions, such as if the assessment is 
based on an as-built or as-designed system, or whether current installed countermeasures 
will be considered. 

(4) Detect and confirm hazards to the system. Identify the targets threatened by each hazard. A 
hazard is defined as an activity or circumstance posing “a potential of loss or harm” to a 
target and is a condition required for an “undesired loss event.” 3 - 3 Hazards should be 
distinguished from consequences and considered in terms of a source (hazard), mechanism 
(process), and outcome (consequence). A team approach to identifying hazards, such as 
brainstorming (sec. 7.7), is recommended over a single analyst. If schedule and resource 
restraints are considerations, then a proficient engineer with knowledge of the system 
should identify the hazards, but that assessment should be reviewed by a peer. A list of 
proven methods* for finding hazards is presented below: 

a. Use intuitive “engineering sense.” 

b. Examine and inspect similar facilities or systems and interview workers assigned to 
those facilities or systems. 

c. Examine system specifications and expectations. 

d. Review codes, regulations, and consensus standards. 

e. Interview current or intended system users or operators. 

f. Consult checklists (app. D). 

g. Review system safety studies from other similar systems. 


^Provided courtesy of Sverdrup Technology, Inc., Tullahoma, Tennessee . 3 3 
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© Identify TARGETS to be protected: © Recognize RISK TOLERANCE LIMITS 

• Personnel . Product • Environment (i- e„ Risk Matrix Boundaries) 

• Equipment • Productivity •...other... 

© "SCOPE* system as to: (a) physical boundaries; (b) oper a ting HAZAR&'^''arCor^po^ Urn* oi Haim 

phases (e. g., shakedown, startup, standard run, emergency stop, 5 Deecribe heart: 
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1 

2 
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ASSESS RISK 


USE RISK MATRIX 
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/es 


■^See© 
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IMPAIR system pertomtance? ...H so, develop NEW COUNTERMEASURES I 


Figure 3-7. PHA Process flowchart 
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h. Review historical documents — mishap files, near-miss reports, OSHA-recordable injury 
rates. National Safety Council data, manufacturer’s reliability analyses, etc. 

i. Consider “external influences” like local weather, environment, or personnel 
tendencies. 

j. Consider all mission phases. 

k. Consider “common causes.” A common cause is a circumstance or environmental 
condition that, if it exists, will induce two or more fault/failure conditions within a 
system. 

l. Brainstorm (sec. 7.7) — mentally develop credible problems and play “what-if ’ games. 

m. Consider all energy sources. What is necessary to keep them under control; what 
happens if they get out of control? 

(5) Assess worst-credible case (not the worst-conceivable case) severity and probability for 
each hazard and target combination. Keep the following considerations in mind during the 
evaluation: 

a. Remember that severity for a specific hazard varies as a function of targets and 
operational phases. 

b. A probability interval must be established before probability can be determined. This 
interval can be in terms of time, or number of cycles or operations. 

c. The assessment will underestimate the true risk if a short-term probability interval is 
used unless the risk acceptance criterion is adjusted accordingly. Probability intervals 
expressed in hours, days, weeks, or months are too brief to be practical. The interval 
should depict the estimated facility, equipment, or each human operator working life 
span. An interval of 25 to 30 yr is typically used and represents a practical value. 

d. The probability for a specific hazard varies as a function of exposure time, target, 
population, and operational phase. 

e. Since probability is determined in a subjective manner, draw on the experience of 
several experts as opposed to a single analyst. 

(6) Assess risk for each hazard using a risk assessment matrix (sec. 3.1). The matrix should be 
consistent with the established probability interval and force or fleet size for this 
assessment. 

(7) Categorize each identified risk as acceptable or unacceptable, or develop countermeasures 
for the risk, if unacceptable. 

(8) Select countermeasures in the following descending priority order to optimize 
effectiveness: (1) “design change,” (2) “engineering safety systems,” (3) “safety devices,” 
(4) “warning devices,” and (5) “procedures and training.” 3 - 3 
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Note that this delineation, while in decreasing order of effectiveness, is also typically in 
decreasing order of cost and schedule impact (i.e., design changes have the highest 
potential for cost and schedule impact). A trade study (sec. 2.1) might be performed to 
determine a countermeasure of adequate effectiveness and minimized program impact. 

(9) Reevaluate the risk with the new countermeasure installed. 

(10) If countermeasures are developed, determine if they introduce new hazards or intolerably 
diminish system performance. If added hazards or degraded performance are unacceptable, 
determine new countermeasures and reevaluate the risk. 


3.2.4 Example 

An example of a completed PHA worksheet 33 for a pressurized chemical intermediate transfer system is 
presented in figure 3-8. (A blank form is included in appendix E.) 



Figure 3-8. Typical PHA. 
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Note that the worksheet from this example contains the following information: 

a. Brief description of the portion of the system, subsystem, or operation covered in the 
analysis. 

b. Declaration of the probability interval. 

c. System number. 

d. Date of analysis. 

e. Hazard (description and identification number). 

f. Hazard targets (check boxes for personnel, equipment, downtime, product 
environment). 

g. Risk assessment before countermeasures are considered; including severity level, 
probability level, and risk priority code (zone from risk matrix, fig. 3-5). 

h. Description of countermeasure (with codes for various types). 

i. Risk assessment after countermeasures are considered, including severity level, 
probability level, and risk priority code. 

j. Signature blocks for the analyst and reviewers/approvers. 

The PHA worksheet used in the example is typical. However, an organization may create 
their own worksheet customized for their operation. For example, different target types may be listed. In 
any case, great care should be given in designing the form to encourage effective usage. Although 
helpful, a PHA is not a structured approach that assists the analyst in identifying hazards or threats. 

3.2.5 Advantages 

A PHA provides the following advantages: 3 3 

(1) Identifies and provides a log of primary system hazards and their corresponding risks. 

(2) Provides a logically based evaluation of a system’s weak points early enough to allow 
design mitigation of risk rather than a procedural or inspection level approach. 

(3) Provides information to management to make decisions to allocate resources and prioritize 
activities to bring risk within acceptable limits. 

(4) Provides a relatively quick review and delineation of the most significant risks associated 
with a specific system. 
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3.2.6 Limitations 


A PHA possesses the following limitations: 33 

(1) A PHA fails to assess risks of combined hazards or coexisting system failure modes. 
Therefore a false conclusion may be made that overall system risk is acceptable simply 
because each hazard element risk identified, when viewed singularly, is acceptable. 

(2) If inappropriate or insufficient targets or operational phases are chosen, the assessment will 
be flawed. While on the other hand, if too many targets or operational phases are chosen, 
the effort will become too large and costly to implement. 


3.2.7 Bibliography 

Air Force Systems Command Design Handbook DH 1-6, “System Safety.” December 1982. 

Army Regulation 3895-16, “System Safety Engineering and Management.” May 1990. 

Browning, R.L.: “The Loss Rate Concept in Safety Engineering.” Marcel Dekker, Inc., 1980. 

Hammer, W.: “Handbook of System and Product Safety.” Prentice-Hall, Inc., 1972. 

Henley, E.J., and Kumamoto, H.: “Probabilistic Risk Assessment.” The Institute of Electrical and 
Electronic Engineers, Inc., New York, 1991. 

Malasky, S.W.: “System Safety: Technology and Application.” Garland STPM Press, 1982. 

Raheja, D.G. “Assurance Technology and Application - Principles and Practices.” McGraw-Hill, 1991. 

Roland, H.E., and Moriaty, B.: “System Safety Engineering and Management.” John Wiley & Sons, 
Inc., 1983. 


3.3 Energy Flow/Barrier Analysis 


3.3.1 Description 

The energy flow/barrier analysis, as described in reference 3.4, is a system safety analysis tool, 
used to identify hazards and determine the effectiveness of countermeasures employed or suggested to 
mitigate the risk induced by these hazards. This tool is also known as energy trace/barrier analysis. The 
energy flow/barrier method is a useful supplement to the PHA discussed in section 3.2. 

Energy sources such as electrical, mechanical, chemical, radiation, etc., are identified. 
Opportunities for undesired energy flow between the sources and targets are assessed. Barriers are 
countermeasures against hazards caused by flows from these energy sources to targets. Examples of 
barriers include barricades, blast walls, fences, lead shields, gloves, safety glasses, procedures, etc. 
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3.3.2 Application 


An energy flow/barrier analysis can be beneficially applied whenever assessments are needed to 
assure an identified target is being safeguarded against a potential energy source that can impose harm. 
This assessment can be applied during phase C but may also be applied in phase E or phase B. This 
analysis can also be applied in failure investigations. 


3.3.3 Procedures 

Procedures to perform an energy flow/barrier analysis, as described in reference 3.4, are 
presented below: 

(1) Examine the system and identify all energy sources. 

(2) Examine each potential energy flow path in the system. Consider the following for each 
energy flow path: 

a. What are the potential targets, such as personnel, facilities, equipment, productivity, 
mission or test objectives, environment, etc.? Remember every energy source could 
have multiple flow paths and targets. 

b. Is the energy flow unwanted or detrimental to a target? 

c. Are existing barriers sufficient countermeasures to mitigate the risk to the targets? 

(3) Consider the following strategies extracted from reference 3.4 to control harmful energy 
flow: 

a. Eliminate energy concentrations. 

b. Limit quantity and/or level of energy. 

c. Prevent the release of energy. 

d. Modify the rate of release of energy. 

e. Separate energy from target in time and/or space. 

f. Isolate by imposing a barrier. 

g. Modify target contact surface or basic structure. 

h. Strengthen potential target. 

i. Control improper energy input. 

3.3.4 Example 

Examples of strategies to manage harmful energy flows are presented in table 3-3. 
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Table 3-3. Examples* of strategies to manage harmful energy flow. 


Strategy 

Examples 

Eliminate energy concentrations 

• Control/limit floor loading 

• Disconnect/remove energy source from system 

• Remove combustibles from welding site 

• Change to nonflammable solvent 

Limit quantity and/or level of energy 

• Store heavy loads on ground floor 

• Lower dam height 

• Reduce system design voltage/operating pressure 

• Use small(er) electrical capacitors/pressure accumulators 

• Reduce/ control vehicle speed 

• Monitor/limit radiation exposure 

• Substitute less energetic chemicals 

Prevent release of energy 

• Heavy-wall pipe or vessels 

• Interlocks 

• Tagout - lockouts 

• Double-walled tankers 

• Wheel chocks 

Modify rate of release of energy 

• Flow restrictors in discharge lines 

• Resistors in discharge circuits 

• Fuses/circuit interrupters 

Separate energy from target in time 
and/or space 

• Evacuate explosive test areas 

• Impose explosives quantity-distance rules 

• Install traffic signals 

• Use yellow no-passing lines on highways 

• Control hazardous operations remotely 

Isolate by imposing a barrier 

• Guard rails 

• Toe boards 

• Hard hats 

• Face shields 

• Machine tool guards 

• Dikes 

• Grounded appliance frames/housing 

• Safety goggles 

Modify target contact surface or basic 
structure 

• Cushioned dashboard 

• Fluted stacks 

• Padded rocket motor test cell interior 

• Whipple plate meteorite shielding 

• Breakaway highway sign supports 

• Foamed runways 

Strengthen potential target 

• Select superior material 

• Substitute forged part for cast part 

• “Harden” control room bunker 

• Cross-brace transmission line tower 

Control improper energy input 

• Use coded, keyed electrical connectors 

• Use match-threaded piping connectors 

• Use back flow preventors 


*Examples provided courtesy of Sverdrup Technology Inc., Tullahoma, Tennessee. 


3-20 




3.3.5 Advantages 


The energy flow/barrier analysis provides a systematic thought process to identify hazards 

associated with energy sources and determines if current or planned barriers are adequate 

countermeasures to protect exposed targets. 3 - 4 

3.3.6 Limitations 

The energy flow/barrier analysis possesses the following limitations. 3 - 4 

(1) Even after a thorough analysis, all hazards might not be discovered. Like the PHA (sec. 
3.2), an energy flow/barrier analysis fails to assess risks of combined hazards or coexisting 
system failure modes. 

(2) This tool also fails to identify certain classes of hazards, e.g., asphyxia in oxygen-deficient 
confined spaces. 

(3) Due to design and performance requirements, it is not always obvious that energy may be 
reduced or redirected. A reexamination of energy as heat, potential versus kinetic 
mechanical energy, electrical, chemical, etc. may aid this thought process. 

3.3.7 Bibliography 

Department of Energy, DOD 76-451: SSDC-29, “Barrier Analysis.” 

Haddon, W., Jr., M.D.: “Energy Damage and the Ten Countermeasure Strategies.” Human Factors 

Journal, August 1973. 

Johnson, W.G.: “MORT Safety Assurance Systems.” Marcel Dekker, Inc., 1980. 


3.4 Failure Modes and Effects (and Criticality) Analysis 


3.4.1 Description 

An FMEA, as described in reference 3.5, is a forward logic (bottom- up), tabular technique that 
explores the ways or modes in which each system element can fail and assesses the consequences of 
each of these failures. In its practical application, its use is often guided by top-down “screening” (as 
described in sec. 3.4.3) to establish the limit of analytical resolution. An FMECA also addresses the 
criticality or risk of individual failures. Countermeasures can be defined, for each failure mode, and 
consequent reductions in risk can be evaluated. FMEA and FMECA are useful tools for cost and benefit 
studies (sec. 2.2), to implement effective risk mitigation and countermeasure, and as precursors to a fault 
tree analysis (FTA) (sec. 3.5). 
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3.4.2 Application 

An FMEA can be used to call attention to system vulnerability to failures of individual compo- 
nents. Single-point failures can be identified. This tool can be used to provide reassurance that the cause, 
effect, and associated risk (FMECA) of component failures have been appropriately addressed. These 
tools are applicable within systems or at the system-subsystem interfaces and can be applied at the 
system, subsystem, component, or part levels. 

These failure mode analyses are typically performed during phase C. During this phase, these 
analyses can be done with or shortly after the PHA (sec. 3.2). The vulnerable points identified in the 
analyses can aid management in making decisions to allocate resources in order to reduce vulnerability. 

3.4.3 Procedures 

Procedures for preparing and performing FMECA’s, as described in reference 3.5, are presented 
below. Procedures for preparing an FMEA are the same with steps 8 through 12 omitted. 

Steps prior to performing the FMEA or FMECA: 

(1) Define the scope and boundaries of the system to be assessed. Gather pertinent information 
relating to the system, such as requirement specifications, descriptions, drawings, compo- 
nents and parts lists, etc. Establish the mission phases to be considered in the analysis. 

(2) Partition and categorize the system into advantageous and reasonable elements to be 
analyzed. These system elements include subsystems, assemblies, subassemblies, 
components, and piece parts. 

(3) Develop a numerical coding system that corresponds to the system breakdown (fig. 3-9). 
Steps in performing the FMEA or FMECA (see flowchart presented in fig. 3-10): 

(4) Identify resources of value, such as personnel, facilities, equipment, productivity, mission 
or test objectives, environment, etc. to be protected. These resources are targets. 

(5) Identify and observe the levels of acceptable risk that have been predetermined and 
approved by management. These limits may be the risk matrix boundaries defined in a risk 
assessment matrix (sec. 3.2). 

(6) By answering the following questions posed in reference 3.5, the scope and resources 
required to perform a classical FMEA can be reduced, without loss of benefit: 

a. Will a system failure render an unacceptable or unwanted loss? 

If the answer is no, the analysis is complete. Document the results. (This has the addi- 
tional benefit of providing visibility of nonvalue added systems, or it may serve to 
correct incomplete criteria being used for the FMEA.) If the answer is yes, ask the 
following question for each subsystem identified in step 2 above. 
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| System 


£ 


Subsystem 1 


I 

Subsystem 2 | Subsystem 3 


Assembly 1 | Assembly 2 Assembly 3 


Subassembly 1 Subassembly 2 iSubassembly 3 


Part 1 


Part 2 


Part 3 


1 


Subsystem 4 


1 

| Compc 

nent 1 

i 

1 

Component 2 


1 

Component 3 


Typical Coding System: Subsystem No. - Assembly No. - Subassembly No. - Component No. - Part No. 
For example, code number for part 2 above is 03-01-03-01 -02 


Figure 3-9. Example of system breakdown and numerical coding. 


3.5 


b. Will a subsystem failure render an unacceptable or unwanted loss? 

If the answer for each subsystem is no, the analysis is complete. Document the results. 
If the answer is yes for any subsystem, ask the following question for each assembly of 
those subsystems identified in step 2 above. 

c. Will an assembly failure render an unacceptable or unwanted loss? 

If the answer for each assembly is no, the analysis is complete. Document the results. If 
the answer is yes for any assembly, ask the following question for each component of 
those assemblies identified in step 2 above. 

d. Will a subassembly failure render an unacceptable or unwanted loss? 

If the answer for each subassembly is no, the analysis is complete. Document the 
results. If the answer is yes for any subassembly, ask the following question for each 
component of those subassemblies identified in step 2 above: 
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Figure 3-10. FMECA process flowchart. 
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e. Will a component failure render an unacceptable or unwanted loss? 

If the answer for each component is no, the analysis is complete. Document the results. 
If the answer is yes for any component, ask the following question for each part of 
those components as identified in step 2 above. 

f. Will a part failure render an unacceptable or unwanted loss? 

(7) For each element (system, subsystem, assembly, subassembly, component, or part) for 
which failure would render an unacceptable or unwanted loss, ask and answer the following 
questions: 

a. What are the failure modes (ways to fail) for this element? 

b. What are the effects (or consequences) of each failure mode on each target? 

(8) Assess worst-credible case (not the worst-conceivable case) severity and probability for 
each failure mode, effect, and target combination. 

(9) Assess risk of each failure mode using a risk assessment matrix (sec. 3.1). The matrix 
should be consistent with the established probability interval and force or fleet size for this 
assessment. 

(10) Categorize each identified risk as acceptable or unacceptable. 

(11) If the risk is unacceptable, then develop countermeasures to mitigate the risk. 

(12) Then reevaluate the risk with the new countermeasure installed. 

(13) If countermeasures are developed, determine if they introduce new hazards or intolerable or 
diminished system performance. If added hazards or degraded performance are unaccept- 
able, develop new countermeasures and reevaluate the risk. 

(14) Document your completed analysis on an FMEA or FMECA worksheet. The contents and 
formats of these worksheets vary among organizations. Countermeasures may or may not 
be listed. Typically, the information found in an FMECA worksheet, according to reference 
3.5, is presented in figure 3-1 1. A worksheet for an FMEA would be similar with the risk 
assessment information removed. A sample FMEA worksheet is attached in appendix F. 


3-25 



FMEA NO: 
PROJECT N 
SUBSYSTE 
SYSTEM N( 
PROB. INTI 

0.: 

FAILURE MODES, EFFECTS, 
AND CRITICALITY ANALYSIS 
WORKSHEET 

SHEE 

DATI 

PREP 

REVI 

APPF 

/r of 

M NO.: 
3.: 

:rval: 

ARED BY: 
EWED BY: 
.OVED BY: 

TARGET CODE: P — PERSONNEL / E — EQUIPMENT / T — DOWNTIME / R - PRODUCTS / D -DATA / V —ENVIRONMENT 

Id. No. 

Item/ 

Functional 

Ident. 

Failure 

Mode 

Failure 

Cause 

Failure 

Event 

T 

a 

r 

g 

e 

t 

Risk Assessment 

Action Required/ 
Comments 

s 

e 

V 

P 

r 

0 

b 

RC 
i o 
s d 
k e 














































































































































Figure 3-11. Typical FMECA worksheet. 3 5 


3.4.4 Example 

An example FMECA 3 5 is illustrated in figure 3-12. The system being assessed is an automated 
mountain climbing rig. A schematic of the system is presented in figure 3-12(<a). Figure 3-12 (b) 
illustrates the breakdown and coding of the system into subsystem, assembly, and subassembly 
elements. An FMECA worksheet for the control subsystem is presented in figure 3- 12(c). 
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(a) System. 


Subsystem 

Assembly 

Subassembly 

Hoist (A) 

Motor (A-01) 

Windings (A-01 -a) 

Inboard bearing (A-01-b) 
Outboard bearing (A-01-c) 
Rotor (A-01 -d) 

Stator (A-01-e) 

Frame (A-01-f) 

Mounting plate (A-01-g) 
Wiring terminals (A-01-h) 

Drum (A-02) 


External power source (B) 



Cage (C) 

Frame (C-01) 
Lifting Lug (C-02) 


Cabling (D) 

Cable (D-01) 
Hook (D-02) 
Pulleys (D-03) 


Controls (E) 

Switch (E-01) 

START (E-01 -a) 

FULL UP LIMIT (E-01-b) 
Wiring (E-01-c) 


(i b ) System breakdown and coding. 


Figure 3-12. Example of an FMECA — Continued 
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p 

E 

T 

IV 

IV 

IV 
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c 

c 

3 

3 

3 


E-01-b 

Full up switch 

Switch fails 
open. 

Mechanical 
failure or 
corrosion. 

Cage does 
not stop. 

P 

II 

A 

1 


E-02 

Wiring 

Cut, dis- 
connected. 

Varmint 

invasion. 

No 

response a 
switch. 

Start switch 
fails open. 
Stop switch 
fails 
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P 
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IV 

IV 

IV 

D 
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(c) Worksheet. 

Figure 3-12. Example of an FMECA — Continued. 


3.4.5 Advantages 

Performing FMEA’s and FMECA’ s provide the following advantages: 3 5 

(1) Provides a mechanism to be exhaustively thorough in identifying potential single-point 
failures and their consequences. An FMECA provides risk assessments of these failures. 

(2) Results can be used to optimize reliability, optimize designs, incorporate “fail safe” 
features into the system design, obtain satisfactory operation using equipment of “low 
reliability,” and guide in component and manufacturer selection. 

(3) Provide further analysis at the piece-part level for high-risk hazards identified in a PHA. 
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(4) Identify hazards caused by failures to be added to the PHA that may have been previously 
overlooked in the PHA. 

(5) Provides a mechanism for more thorough analysis than an FTA, since every failure mode of 
each component of the system is assessed. 3 6 


3.4.6 Limitations 


The following limitations are imposed when performing FMEA’s and FMECA’s. 

(1) Costly in manhour resources, especially when performed at the parts-count level within 
large, complex systems. 

(2) Probabilities or the consequences of system failures induced by coexisting, multiple- 
element faults or failures within the system are not addressed or evaluated. 

(3) Although systematic, and guidelines/check sheets are available for assistance, no check 
methodology exists to evaluate the degree of completeness of the analysis. 

(4) This analysis is heavily dependent upon the ability and expertise of the analyst for finding 
all necessary modes. 

(5) Human error and hostile environments frequently are overlooked. 

(6) Failure probability data are often difficult to obtain for an FMECA. 

(7) If too much emphasis is placed on identifying and eliminating single-point failures, then 
focus on more severe system threats may be overlooked. 

(8) An FMECA can be a very thorough analysis suitable for prioritizing resources to higher 
risk areas if it can be performed early enough in the design phase. However, the level of 
design maturity required for an FMECA is not generally achieved until late in the design 
phase, often too late to guide this prioritization. 
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3.5 Reliability Block Diagram 


3.5.1 Description 

A reliability block diagram (RBD) is a backwards (top-down) symbolic logic model generated in 
the success domain. The descriptions of RDB’s contained herein was obtained from references 3.7 and 
3.8. Each RBD has an input and an output and flows left to right from the input to the output. Blocks 
may depict the events or system element functions within a system. However, these blocks typically 
depict system element functions only. A system element can be a subsystem, subassembly, component, 
or part. 


Simple RBD’s are constructed of series, parallel, or combinations of series and parallel elements 
(table 3-4). Each block represents an event or system element function. These blocks are connected in 
series if all elements must operate successfully for the system to operate successfully. These blocks are 
connected in parallel if only one element needs to operate successfully for the system to operate success- 
fully. A diagram may contain a combination of series and parallel branches. The system operates if an 
uninterrupted path exists between the input and output. 


Table 3-4 . Simple RBD construction. 


Type branch 


Block diagram representation 


System reliability # 


Series 


B 


Rs = Ra* Rb 


Parallel 


A 


B 


R S = 1 -(\-R a )(\-R b ) 


Series-parallel 



1 — 

A 

— 1 


i 

c 

— i 













1 

B 

H 


i 

D 

H 



R s =( 1-(1-R A) (1-R b )) 
*(1 - ( 1 -R C) ( 1 -Ro)) 


Parallel-series 



1 

A 


B 

“1 











1 

C 


D 

H 



R s =1-(1-(R A *R B )) 
* (1 - (Re* r d)) 


^Assumes all components function independently of each other. 
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RBD’s illustrate system reliability. Reliability is the probability of successful operation during a 
defined time interval. Each element of a block diagram is assumed to function (operate successfully) or 
fail independently of each other element. The relationships between element reliability and system relia- 
bility for series and parallel systems are presented below, and their derivations are found in reference 
3.8. 

Series Systems: 

n 

Rs = n Ri= R\* * ^3 * ' ' ' ' R }1 - 


Parallel Systems: 

r s = i-n = [l-a-flt )*(i— /?2 ) (1-^3 )*•••• (i-Rn u 

i 

where 

Rs = system reliability, 

Ri = system element reliability, and 

n = number of system elements (which are assumed to function independently). 

Not all systems can be modeled with simple RBD’s. Some complex systems cannot be modeled 
with true series and parallel branches. These systems must be modeled with a complex RBD. Such an 
RBD is presented in figure 3-13. Notice in this example, if element E fails, then paths B, E, G and B, E, 
H are not success paths, thus this is not a true series or parallel arrangement. 
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3.5.2 Application 


An RBD allows evaluation of various potential design configurations. 3 - 8 Required subsystem and 
element reliability levels to achieve the desired system reliability can be determined. Typically, these 
functions are performed during phase C. An RBD may also be used to identify elements and logic as a 
precursor to performing an FTA (sec. 3.6). 


3.5.3 Procedures 

The procedures (adapted from reference 3.8) to generate a simple RBD are presented below: 

(1) Divide a system into its elements. A functional diagram of the system is helpful. 

(2) Construct a block diagram using the convention illustrated in table 3-4. 

(3) Calculate system reliability band, Rsl (low) to Rs/i (high), from each individual element’s 
reliability band, Rn (low) to 7^/// (high), in the following manner: 

a. For series systems with n elements that are to function independently, 

Rsl = n (Rn) = Ril ■■■Ril *R3l * — RnL 


n 

RsH= n (Rill) = RlH*R 2 H *R3H * RrH. 


b. For parallel systems with n elements that are to function independently, 


Rsl = 


n 

i-n 


i 


(1 -RpL) = [1-(1-Riz>(1-R 2 l) (1 -R3L )*•••• (1-R»l)] 


rsh = i-n 


i 


(1 -R p h) = [1-(1-Rih)*(1-R2H) ( \-R3H)* (1 -R?iH)]- 


c. For series-parallel systems, first determine the reliability for each parallel branch using 
the equations in step 3b. Then treat each parallel branch as an element in a series branch 
and determine the system reliability by using the equations in step 3a. 

d. For parallel-series systems, first determine the reliability for each series branch using 
the equations in step 3a. Then treat each series branch as an element in a parallel branch 
and determine the system reliability by using the equations in step 3b. 
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e. For systems that are composed of the four above arrangements, determine the reliability 
for the simplest branches. Then, treat these as branches within the remaining block dia- 
gram, and determine the reliability for the new simplest branches. Continue this process 
until one of the above four basic arrangements remains, then determine the system 
reliability. 


3.5.4 Example 

A system has two subsystems designated 1 and 2. Subsystem 2 is designed to be a backup for 
subsystem 1. Subsystem 1 has three components and at least one of the three must function successfully 
for the subsystem to operate. Subsystem 2 has three components that all need to function successfully 
for the subsystem to operate. The estimated reliability band for each individual component over the 
system’s estimated 10-yr life interval is presented below: 


Subsystem 

Component 

Reliabilitv Bands 
Low High 

1 

A 

0.70 

0.72 

1 

B 

0.80 

0.84 

1 

C 

0.60 

0.62 

2 

D 

0.98 

0.99 

2 

E 

0.96 

0.97 

2 

F 

0.98 

0.99 


An RBD for the system is presented in figure 3-14. Note that the components for subsystem 1 are 
in a parallel branch with the components of subsystem 2. Also, note that the components for subsystem 1 
form a series branch and the components for subsystem 2 form a parallel branch. 


0.70 - 0.72 



Figure 3-14. Example RBD. 
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C alculations for subsystem and system reliabilities are presented below: 


Subsystem 1: 

RlL = 1 — (1— 0.70)(1— 0.80)(1— 0.60) = 0.976 

(low band value) 


R iH = 1 - ( 1— 0.72)( 1— 0.84)(1— 0.62) = 0.983 

(high band value) 

Subsystem 2: 

r 2l = (0.98)(0.96)(0.98) = 0.922 

(low band value) 


R 2H = (0.99)(0.97)(0.99) = 0.951 

(high band value) 

System: 

R S l= 1 - (1-0.976X1-0.922) = 0.998 

(low band value) 


R SH = 1 - (1-0.983)(1-0.951) = 0.999 

(high band value) 


Therefore, the reliability band for the system is 0.998 to 0.999. 


3.5.5 Advantages 

An RBD provides the following advantages: 

(1) Allows early assessment of design concepts when design changes can be readily and 
economically incorporated. 3 8 

(2) Tends to be easier for an analyst to visualize than other logic models, such as a fault tree. 3 - 7 

(3) Blocks representing elements in an RBD can be arranged in a manner that represent how 
these elements function in the system. 3 - 7 

(4) Since RBD’s are easy to visualize, they can be generated prior to performing an FTA and 
transformed into a fault tree by the method discussed in section 3.9. 


3.5.6 Limitations 


An RBD possesses the following limitations: 

(1) Systems must be broken down into elements where reliability estimates can be obtained. 
Such a breakdown for a large system can be a significant effort. 3 8 

(2) System element reliability estimates might not be readily available for all elements. Some 
reliability estimates may be very subjective, difficult to validate, and not be accepted by 
others in the decision making process. If the element reliability values have different 
confidence bands, this can lead to significant problems. 

(3) Not all systems can be modeled with combinations of series, parallel, series-parallel, or 
parallel- series branches. These complex systems can be modeled with a complex RBD. 
However, determining system reliability for such a system is more difficult than for a 
simple RBD. 3 73 8 
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3.6 Fault Tree Analysis 


3.6.1 Description 

A fault tree analysis (FTA), as described in reference 3.9, is a top-down symbolic logic model 
generated in the failure domain. This model traces the failure pathways from a predetermined, 
undesirable condition or event, called the TOP event, of a system to the failures or faults (fault tree 
initiators) that could act as causal agents. Previous identification of the undesirable event also includes a 
recognition of its severity. An FTA can be carried out either quantitatively or subjectively. 

The FTA includes generating a fault tree (symbolic logic model), entering failure probabilities 
for each fault tree initiator, propagating failure probabilities to determining the TOP event failure 
probability, and determining cut sets and path sets. A cut set is any group of initiators that will, if they 
all occur, cause the TOP event to occur. A minimal cut is a least group of initiators that will, if they all 
occur, cause the TOP event to occur. A path set is a group of fault tree initiators that, if none of them 
occurs, will guarantee the TOP event cannot occur. 

The probability of failure for a given event is defined as the number of failures per number of 
attempts. This can be expressed as: 

Pf = F/(S+F ) , where F = number of failures and S = number of successes. 

Since reliability for a given event is defined as the number of successes per number of attempts, 
then the relationship between the probability of failure and reliability can be expressed as follows: 

R = S/(S+F), 


therefore 


R + P f = S/(S+F) + F/(S+F) = 1 


and 


P F = 1 -R. 


3.6.2 Application 

FTA’s are particularly useful for high energy systems (i.e., potentially high severity events), to 
ensure that an ensemble of countermeasures adequately suppresses the probability of mishaps. An FTA 
is a powerful diagnostic tool for analysis of complex systems and is used as an aid for design 
improvement. 
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This type of analysis is sometimes useful in mishap investigations to determine cause or to rank 
potential causes. Action items resulting from the investigation may be numerically coded to the fault tree 
elements they address, and resources prioritized by the perceived highest probability elements. 

FTA’s are applicable both to hardware and nonhardware systems and allow probabilistic assess- 
ment of system risk as well as prioritization of the effort based upon root cause evaluation. The 
subjective nature of risk assessment is relegated to the lowest level (root causes of effects) in this study 
rather than at the top level. Sensitivity studies can be performed allowing assessment of the sensitivity of 
the TOP event to basic initiator probabilities. 

FTA’s are typically performed in phase C but may also be performed in phase D. FTA’s can be 
used to identify cut sets and initiators with relatively high failure probabilities. Therefore, deployment of 
resources to mitigate risk of high-risk TOP events can be optimized. 


3.6.3 Procedures 


The procedures, as described in reference 3.9, for performing an FTA are presented below. These 
procedures are divided into the four phases: (1) fault tree generation, (2) probability determination, 

(3) identifying and assessing cut sets, and (4) identifying path sets. The analyst does not have to perform 
all four phases, but can progress through the phases until the specific analysis objectives are met. The 
benefits for each of the four phases are summarized in table 3-5. 


Table 3-5. FTA procedures. 


Section 

Procedures 

Benefits 

3.6.3. 1 

Fault tree generation 

All basic events (initiators), intermediate events, and the TOP 
event are identified. A symbolic logic model illustrating fault 
propagation to the TOP event is produced. 

3.6. 3. 2 

Probability determination 

Probabilities are identified for each initiator and propagated to 
intermediate events and the TOP event. 

3.6. 3. 3 

Identifying and assessing cut sets 

All cut sets and minimal cuts sets are determined. A cut set is 
any group of initiators that will, if they all occur, cause the 
TOP event to occur. A minimal cut set is a least group of 
initiators that, if they all occur, will cause the TOP event to 
occur. Analysis of a cut set can help evaluate the probability 
of the TOP event, identify qualitative common cause 
vulnerability, and assess quantitative common cause 
probability. Cut sets also enable analyzing structural, 
quantitative, and item significance of the tree. 

3.6. 3.4 

Identifying path sets 

All path sets are determined. A path set is a group of fault tree 
initiators that, if none of them occurs, will guarantee the TOP 
event cannot occur. 
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3.6.3. 1 Fault Tree Generation 


Fault trees are constructed with various event and gate logic symbols, defined in table 3-6. 
Although many event and gate symbols exist, most fault trees can be constructed with the following four 
symbols: (1) TOP or Intermediate event, (2) inclusive OR gate, (3) AND gate, and (4) basic event. The 
procedures, as described in reference 3.9, to construct a fault tree are illustrated in figure 3-15. 

A frequent error in fault tree construction is neglecting to identify common causes. A common 
cause is a condition, event, or phenomenon that will simultaneously induce two or more elements of the 
fault tree to occur. A method for detecting common causes is described in section 3. 6. 3. 3, step 8. 
Sections 3. 6. 3. 2 through 3. 6. 4. 3 are included for completeness and to provide insight as to the 
mathematics that takes place in the commercially available fault tree programs. All large trees are 
typically analyzed using these programs; for small trees hand analysis may be practical. 

3. 6. 3. 2 Probability Determination 

If a fault tree is to be used as a quantitative tool, the probability of failure must be determined for 
each basic event or initiator. Sources for these failure probabilities may be found from manufacturer’s 
data, industry consensus standards, MIL- standards, historical evidence (of the same or similar systems), 
simulation or testing, Delphi estimates, and the log average method. A source for human error probabili- 
ties is found in reference 3.10. The Delphi technique (sec. 7.9) derives estimates from the consensus of 
experts. The log average method is useful when the failure probability cannot be estimated but credible 
upper and lower boundaries can be estimated. This technique is described in reference 3.11 and is 
illustrated in figure 3-16. 

Failure probabilities can also be determined from a PDA as discussed in section 3.14.3, step 6. 

Probabilities must be used with caution to avoid the loss of credibility of the analysis. In many 
cases it is best to stay with comparative probabilities rather than the “absolute” values. Normalizing data 
to a standard, explicitly declared meaningless value is a useful technique here. Also, confidence or error 
bands, on each cited probability number, are required to determine the significance of any quantitatively 
driven conclusion. 

Once probabilities are estimated for all basic events or initiators, they are propagated through 
logic gates to the intermediate events and finally the TOP event. The probability of failure of 
independent inputs through an AND gate is the intersection of their respective individual probabilities. 
The probability of failure of independent events through an OR (inclusive) gate is the union of their 
respective individual probabilities. Propagation of confidence and error bands is performed simply by 
propagation of minimum and maximum values within the tree. 

The relationship between reliability and failure probability propagation of two and three inputs 
through OR (inclusive) and AND gates is illustrated in figure 3-17. Propagation of failure probabilities 
for two independent inputs through an AND and OR (inclusive) is conceptually illustrated in figure 3- 
18. As shown in figures 3-17, the propagation solution through an OR gate is simplified by the rare 
event approximation assumption. The exact solution for OR gate propagation is presented in figure 3-19. 
However, the use of this exact solution is seldom warranted. 

The propagation equations for the logic gates, including the gates infrequently used, are 
presented in table 3-7. 
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Table 3-6. Fault tree construction symbols. 


Symbol 

Name 

Description 


] 

Event (TOP or 
intermediate) 

TOP Event - This is the conceivable, undesired event to which 
failure paths of lower level events lead. 

Intermediate Event - This event describes a system condition 
produced by preceding events. 

L 


Inclusive OR gate 

An output occurs if one or more inputs exist. Any single input is 
necessary and sufficient to cause the output event to single occur. 

6. 

js 

Exclusive OR gate 

An output occurs if one, but only one input exists. Any single 
input is necessary and sufficient to cause the output event to 
occur. 

L 

\ 

T 

Mutually exclusive 
OR gate 

An output occurs if one or more inputs exist. However, all other 
inputs are then precluded. Any single input is necessary and 
sufficient to cause the output event to occur. 

/ 

“H 


AND gate* 

An output occurs if all inputs exist. All inputs are necessary and 
sufficient to cause the output event to occur. 

i 

§ 


Priority AND gate 

An output occurs if all inputs exist and occur in a predetermined 
sequence. All inputs are necessary and sufficient to cause the 
output event to occur. 

/ 

\ 

, ^ 


INHIBIT gate 

An output occurs if a single input event occurs in presence of an 
enabling condition. 

( 

3 


Basic event 

An initiating fault or failure that is not developed further. These 
events determine the resolution limit of the analysis. They are 
also called leaves or initiators. 

y 



External event 

An event that under normal conditions is expected to occur. 

o 

Undeveloped event 

An event not further developed due to a lack of need, resources, 
or information. 

C 

) 

Conditioning Event 

These symbols are used to affix conditions, restraints, or 
restrictions to other events. 


*Most fault trees can be constructed with these four logic symbols. 
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Basic Event. ..(“Leaf,” 
“Initiator,” or “Basic”) 
indicates limit of analytical 
resolution. 


1. Identify undesirable TOP 
event. 

3. Link contributors to TOP 
by logic gates. 

2. Identify first-level 
contributors. 

5. Link second-level 

contributors to TOP by 
logic gates. 


Identify second-level 
contributors. 


6. Repeat / 
continue... 


Figure 3-15. Fault tree construction process. 


• Estimate upper and lower credible bounds of probability for the phenomenon in question. 

• Average the logarithms of the upper and lower bounds. 

• The antilogarithm of the average of the logarithms of the upper and lower bounds is less 
than the upper bound and greater than the lower bound by the same factor. Thus, it is 
geometrically midway between the limits of estimation. 

0.01 0.02 0.03 0.04 0.05 0.07 0.1 



Note that, for the example shown, the arithmetic average would be... 

0.01 + 0.1 = 0.055 

2 

i.e., 5.5 times the lower bound and 0.55 times the upper bound. 

Figure 3-16. Log average method of probability estimation. 
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OR Gate I For 2 Inputs AND Gate 
Either of two, independent, Both of two, independent 

element failures produces elements must fail to produce 

system failure. ip + p TJI system failure. 

R = R R R = R.+R -RR 


P p = 1 — (R a R b ) 

Rp = 1 - K 1 - p B A )(i-P B )1 

P = P +P — P P [Union/ u] 

F A B A Bl U 


P f = 1 — (R a + R b - R a R b ) 

p F =1 - [Cl - p A ) + (1 - P J - (1 - P J(1 - P J] 


P = P P 

F A B 


[Intersection / n] 



Figure 3-17. Relationship between reliability and failure probability propagation. 



Figure 3-18. Failure probability propagation through OR and AND gates. 


3-40 











The ip operator ( U ) is the 

co-function of pi ( FI )■ It 

provides an exact solution 
for propagating probabilities 
through the OR gate. Its use 
is rarely justifiable. 


P T = U p e =1-P (1-Pe ) 

P T = 1 - [(1 - P l)( 1-P 2 ) (1 -P 3 ) —(1 - Pn)] 


Figure 3-19. Exact solution of OR gate failure probability propagation. 


3. 6. 3. 3 Identifying and Assessing Cut Sets 

A cut set is any group of initiators that will produce the TOP event, if all the initiators in the 
group occur. A minimal cut set is the smallest number (in terms of elements, not probability) of initiators 
that will produce the TOP event, if all the initiators in the group occur. One method of determining and 
analyzing cut sets is presented below. These procedures for determining cut sets are described in 
reference 3.9 and are based on the MOCUS computer algorithm attributed to J.B. Fussell. Analysis of a 
cut set can help evaluate the probability of the TOP event, identify common cause vulnerability, and 
assess common cause probability. Cut sets also enable analyzing structural, quantitative, and item 
significance of the tree. 

Determining Cut Sets: 

(1) Consider only the basic events or initiators (discarding intermediate events and the TOP 
event). 

(2) Assign a unique letter to each gate and a unique number to each initiator, starting from the 
top of the tree. 

(3) From the top of the tree downwards, create a matrix using the letters and numbers. The 
letter for the gate directly beneath the TOP event will be the first entry in the matrix. 
Proceed through the matrix construction by (1) substituting the letters for each AND gate 
with letters for the gates and numbers of the initiators that input into that gate (arrange 
these letters and numbers horizontally in the matrix rows), and (2) substituting the letters 
for each OR gate with letters for the gates and numbers of the initiators that input into that 
gate (arrange these letters and numbers vertically in the matrix columns). 
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Table 3-7. Probability propagation expressions for logic gates. 


Symbol 

Name 

Venn Diagram 

Propagation Expressions 

4 

1 

Inclusive OR 
gate^ 

CD 

PI P2 

P T = p x +P 2 - (Pi * P'2) 
Pj— P\ + P 2 # 

L 

\ 

Exclusive OR 
gate 

GO 

PI P2 

Pt=P\ + P 2 -2 (Pi * P 2 ) 
Pj — P\ + P^fi 

A 

Mutually 
exclusive OR 
gate 

OO 

PI P2 

Pj = P 1 + P 2 

> 

"N 


AND gate $ and 
(priority AND 
gate) 

GO 

PI P2 

P T = Pi * P 2 


•i'Most fault trees can be constructed with these two logic gates. 
^Simplified expression for rare event approximation assumption. 


(4) When all the gate’s letters have been replaced, a final matrix is produced with only 
numbers of initiators. Each row of this matrix represents a Boolean-indicated cut set. 

(5) Visually inspect the final matrix and eliminate any row that contains all elements of a lesser 
row. Next, through visual inspection, eliminate redundant elements within rows and rows 
that repeat other rows. The remaining rows define the minimal cut sets of the fault tree. 

Assessing Cut Sets: 

(6) Since a cut set is any group of initiators that will produce the TOP event, if all the initiators 
in the group occur, the cut set probability, P K (the probability that the cut set will induce the 
TOP event) is mathematically the same as the propagation through an AND gate, expressed 
as: 


Pk = P\*P2* P3* Pa ■■■* Pn- 
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(7) Determine common cause vulnerability by uniquely assigning letter subscripts for common 
causes to each numbered initiator (such as m for moisture, h for human operator, q for heat, 
v for vibration, etc.). Note that some initiators may have more than one subscript, while 
others will have none. Check to see if any minimal cut sets have elements with identical 
subscripts. If that is the case, then the TOP event is vulnerable to the common cause the 
subscript represents. This indicates that the probability number, calculated as above, may 
be significantly in error, since the same event (the so-called common cause) could act to 
precipitate each event, i.e., they no longer represent statistically independent events. 

(8) Analyze the probability of each common cause at its individual probability level of both 
occurring, and inducing all terms within the affected cut set. 

(9) Assess the structural significance of the cut sets to provide qualitative ranking of contribu- 
tions to system failure. Assuming all other things are equal then: 

a. A cut set with many elements indicates low vulnerability. 

b. A cut set with few elements indicates high vulnerability. 

c. Numerous cut sets indicates high vulnerability. 

d. A cut set with a single initiator, called a singleton, indicates a potential single -point 
failure. 

(10) Assess the quantitative Importance, I x, of each cut set, K. That is, determine the numerical 
probability that this cut set induced the TOP event, assuming it has occurred. 

Ik = Pk/Pt 

where 

Pk= the probability that the cut set will occur (see step 6 above), and 
Pj = the probability of the TOP event occurring. 

(1 1) Assess the quantitative importance, I e of each individual initiator, e. That is, determine the 
numerical probability that initiator e contributed to the TOP event, if it has occurred: 

N e 

X 

e 

where 


N e = number of minimal cut sets containing initiator e, and 
1% = importance of the minimal cut sets containing initiator e. 

3. 6. 3.4 Identifying Path Sets 

A path set is a group of fault tree initiators that, if none of them occurs, ensures the TOP event 
cannot occur. Path sets can be used to transform a fault tree into a reliability diagram (sec. 3.9). The 
procedures to determine path sets are as follows: 

(1) Exchange all AND gates for OR gates and all OR gates for AND gates on the fault tree. 
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(2) Construct a matrix in the same manner as for cut sets (sec. 3. 6. 3. 3, steps 1-5). 
Each row of the final matrix defines a path set of the original fault tree. 


3.6.4 Examples 

3.6.4. 1 Fault Tree Construction and Probability Propagation 

An example of a fault tree with probabilities propagated to the TOP event is presented in 
figure 3-20. In this example the TOP event is the “artificial wakeup fails.” The system being examined 
consists of alarm clocks used to awaken someone. In this example for brevity, only a nominal 
probability value for each fault tree initiator is propagated through the fault tree to the TOP event. 
However, for a thorough analysis, both low and high probability values that define a probability band for 
each initiator could be propagated through the fault tree to determine a probability band for the TOP 
event. 



4x1 O' 4 2x10 4 


1/10 1/20 


Figure 3-20. Example fault tree. 
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3. 6. 4. 2 Cut Sets 


An example of how to determine Boolean-indicated minimal cut sets for a fault tree is presented 
in figure 3-21. 


PROCEDURE: 

• Assign letters to 
gates. (TOP gate 
is “A.”) Do not 
repeat letters. 

• Assign numbers 
to basic initiators. 
If a basic appears 
more than once, 
represent it by the 
same number at 
each appearance. 



• Construct a matrix, starting with the TOP “A” gate... 



a 


TOP event gate 
is A, the 
initial matrix 
entry. 


A is an AND 
gate. B& D, 
its inputs, re- 
place it hori- 
zontally. 


B is an OR 
gate. 1 & C, 
its inputs, re- 
place it verti- 
cally. Each 
requires a new 
row. 


C is an AND 
gate. 2 & 3, 
its inputs, 
replace it 
horizontally. 



D (top row), is 
an OR gate. D 
2 & 4, its in- 
puts, replace 
it vertically. 

Each requires a 
new row. 


1 

2 


_2_ 

_2_ 

JL 

1 

4 


2 

4 

3 


These Boolean-Indicated 
Cutsets... 


(2nd row), is 
an OR gate. 
Replace as 
before. 


...reduce to 
these Minimal 
Cut Sets. 



Minimal Cut Set 
rows are least 
groups of 
initiators which 
■.will induce TOP./ 


Figure 3-21. Example of determining cut sets. 
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3. 6. 4. 3 Path Sets 


An example of how to determine path sets for a fault tree is presented in figure 3-22. 


TOP 



Tree 

has... 


1 

2 



1 

3 



1 

4 



3 

4 

5 

6 



...these Minimal 
Cut Sets 




...and these Path 
Sets. 


Path Sets are 
least groups of 
initiators which, if 
they cannot 
occur, guarantee 
against TOP 
occurring. 


1 

3 


1 

4 


1 

5 


T 

TT 


2 

3 

T 


“Barring" terms (n) denotes consideration 
of their success properties. 


Figure 3-22. Example of determining path sets. 


3.6.5 Advantages 

An FTA provides the following advantages: 3 9 

(1) Enables assessment of probabilities of combined faults/failures within a complex system. 

(2) Single-point and common cause failures can be identified and assessed. 

(3) System vulnerability and low-payoff countermeasures are identified, thereby guiding 
deployment of resources for improved control of risk. 

(4) This tool can be used to reconfigure a system to reduce vulnerability. 

(5) Path sets can be used in trade studies to compare reduced failure probabilities with 
increases in cost to implement countermeasures. 
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3.6.6 Limitations 


A FTA possesses the following limitations: 

(1) Address only one undesirable condition or event that must be foreseen by the analyst. Thus, 
several or many FTA’s may be needed for a particular system. 

(2) Fault trees used for probabilistic assessment of large systems may not fit or run on 
conventional PC-based software. 

(3) The generation of an accurate probabilistic assessment may require significant time and 
resources. Caution must be taken not to “over work” determining probabilities or 
evaluating the system, i.e., limit the size of the tree. 

(4) A fault tree is not accurate unless all significant contributors of faults or failures are 
anticipated. 

(5) Events or conditions under the same logic gate must be independent of each other. 

(6) A fault tree is flawed if common causes have not been identified. 

(7) Events or conditions at any level of the tree must be independent and immediate 
contributors to the next level event or condition. 

(8) The failure rate of each initiator must be constant and predictable. Specific 
(noncomparative) estimates of failure probabilities are typically difficult to find, to achieve 
agreement on, and to successfully use to drive conclusions. Comparative analyses are 
typically as valuable with better receptions from the program and design teams. 
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3.7 Success Tree Analysis 


3.7.1 Description 

A success tree analysis (STA) is a backwards (top-down) symbolic logic model generated in the 
success domain. This model traces the success pathways from a predetermined, desirable condition or 
event (TOP event) of a system to the successes (success tree initiators) that could act as causal agents. 
An STA is the compliment of an FTA (sec. 3.6) which is generated in the failure domain with failure 
pathways from undesirable events. 

The STA includes generating a success tree (symbolic logic model), determining success proba- 
bilities for each tree initiator, propagating each initiator probability to determining the TOP event 
probability and determining cut sets and path sets. In the success domain, a cut set is any group of 
initiators that will, if they all occur, prevent the TOP event from occurring. A minimal cut set is a least 
group of initiators that will, if they all occur, prevent the TOP event from occurring. A path set is a 
group of success tree initiators that, if all of them occur, will guarantee the TOP event occurs. 

The probability of success for a given event is defined as the number of successes per number of 
attempts. This can be expressed as: 

P s = S/(S+F ) , where S = number of successes and F = number of failures. 

Since reliability for a given event is also defined as the number of successes per number of 
attempts, then 


R = P S . 


3.7.2 Application 

Particularly useful for high energy systems (i.e., potentially high severity events), to ensure that 
an ensemble of countermeasures adequately leads to a successful top event. This technique is a powerful 
diagnostic tool for analysis of complex systems and is used as an aid for design improvement and is 
applicable both to hardware and nonhardware systems. This technique also allows probabilistic assess- 
ment of causal benefits as well as prioritization of effort based upon root cause evaluation. The 
subjective nature of the probability assessment is relegated to the lowest level (root causes of effects) in 
this study rather than at top level. Sensitivity studies can be performed allowing assessment of the 
sensitivity of study results to subjective numbers. 
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The STA is typically applied in phase C but may also be applied in phase D. A success tree can 
be used to verify the logic of a fault tree. Since a success tree is the logic compliment of a fault tree, if a 
success tree is generated from a fault tree, the logic of the success tree needs to be valid if the logic of a 
fault tree is to be valid. 


3.7.3 Procedures 


Success trees, like fault trees, are constructed with various event and gate logic symbols. These 
symbols are defined in table 3-6. Although many event and gate symbols exist, most success trees can be 
constructed with the following four symbols: (1) TOP or intermediate event, (2) inclusive OR gate, (3) 
AND gate, and (4) basic event. The procedures, as described in reference 3.9, to construct a fault tree 
also apply to success tree generation and are illustrated in figure 3-23. The commercial computer 
programs are similar, as are the cautions for use of probability values. 



Identify desirable TOP 
event. 

Link contributors to TOP 
by logic gates. 



Basic Event. ..(“Leaf,” 
“Initiator," or “Basic”) 
indicates limit of analytical 
resolution. 



2. Identify first-level 
contributors. 

Link second-level 
contributors to TOP by 
logic gates. 


Identify second-level 
contributors. 

Repeat / continue... 


Figure 3-23. Success tree construction process. 


A success tree can be constructed from a fault tree. Transform a success tree from a fault tree by 
simply changing all AND gates to OR gates and OR gates to AND gates, and restating each initiator, 
intermediate event, and top event as a success opposed to a failure. 

Determine the probability of success (P s ) for each basic event or initiator. Sources for these suc- 
cess probabilities may be found from manufacturer’s data, industry consensus standards, MIL standards, 
historical evidence (of similar systems), simulation or testing. Delphi estimates, and the log average 
method. The Delphi technique (sec. 7.9) derives estimates from the consensus of experts. Remember that 
the probability of success equals reliability (R) and may be determined from (Pp) as shown in the 
following equation: 


Ps=l~P F . 


Once probabilities are estimated for all basic events or initiators, propagate these probabilities 
through logic gates to the intermediate events and finally the TOP event. Use the expressions presented 
in table 3-7 to propagate probabilities through logic gates. 
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Generate cut sets and path sets in the same manner as for fault trees, as presented in sections 
3.7. 3. 3 and 3. 7. 3. 4, respectively. 

3.7.4 Example 

The compliment success tree for the fault tree presented in section 3.6.4 is presented in figure 3- 
24. 



0.9998 0.9996 

Figure 3-24. Example success tree. 
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3.7.5 Advantages 

An STA provides the following advantages: 3 9 

(1) Assesses probability of favorable outcome of system operation. 

(2) Compliments the FT A by providing a method to verify the logic of the fault tree. 

3.7.6 Limitations 

An STA possesses the following limitations: 3 9 

(1) Address only one desirable condition or event that must be foreseen by the analyst. Thus, 
several or many STA’s may be needed for a particular system. 

(2) Success trees used for probabilistic assessment of large systems may not fit/run on 
conventional PC-based software. 

(3) The generation of an accurate probabilistic assessment may require significant time and 
resources. Caution must be taken not to overdo the number generation portion. 

(4) A success tree is not accurate unless all significant contributors to system successes are 
anticipated. 

(5) Events or conditions under the same logic gate must be independent of each other. 

(6) Events or conditions at any level of the tree must be independent and immediate 
contributors to the next level event or condition. 

(7) The probability of success (reliability) of each initiator must be constant and predictable. 

3.7.7 Bibliography 
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3.8 Event Tree Analysis 


3.8.1 Description 

An event tree analysis (ETA), as described in references 3.6 and 3.12, is a forward (bottom-up) 
symbolic logic modeling technique generated in both the success and failure domain. This technique 
explores system responses to an initiating “challenge” and enables assessment of the probability of an 
unfavorable or favorable outcome. The system challenge may be a failure or fault, an undesirable event, 
or a normal system operating command. 
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A generic event tree portrays all plausible system operating alternate paths from the initiating 
event. A generic event tree is illustrated in figure 3-25. A Bernoulli model event tree uses binary 
branching to illustrate that the system either succeeds or fails at each system logic branching node. A 
Bernoulli model event tree is illustrated in figure 3-26. A decision tree is a specialized event tree with 
unity probability for the system outcome. 
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Figure 3-25. Event tree (generic case). 


3.8.2 Application 

The ETA is particularly useful in analyzing command-start or command-stop protective devices, 
emergency response systems, and engineered safety features. The technique is useful in evaluating 
operating procedures, management decision options, and other nonhardware systems. The ETA is also 
useful in evaluating effect and benefit of subtiered or redundant design countermeasures for design 
trades and assessment. 

An ETA may be used in conjunction with an FTA to provide a technique sensitivity assessment. 
However, success or failure probabilities used must be used with caution to avoid the loss of credibility 
of the analysis. In many cases it is best to stay with comparative probabilities rather than the “absolute” 
values. Normalizing data to a standard, explicitly declared meaningless value is a useful technique here. 
Also, confidence or error bands, on each cited probability number, are required to determine the signifi- 
cance of any quantitatively driven conclusion. 

An ETA may also be performed to compliment an FMEA. This technique is typically performed 
in phase C or E but may also be performed in phase D. 
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Figure 3-26. Event tree (Bernoulli model). 


3.8.3 Procedures 


The procedures, as described in reference 3.12, for performing an ETA are presented below. 

(1) Identify the initiating challenge to the system being examined. 

(2) Determine the paths (alternate logic sequences) by answering the question, “What happens 
when the system is challenged by the initiation event?” By convention, trace successful 
paths upwards and failure paths downwards. 

a. For the general event tree, trace all plausible system operating permutations to a success 
or failure termination. 

b. For the Bernoulli model event tree, use binary branching to show the system pathways. 
Simplify the tree by pruning unnecessary alternate branches of nonrecoverable failures 
or undefeatable successes. 

(3) Determine the probability of the initiating event by applying a fault tree (sec. 3.6) or other 
analysis. For a decision tree, assume the probability of the initiating event is one. 

(4) Determine the probability of each potential path by multiplying the individual probabilities 
of events making up the path. 

(5) Determine the probability of the system success by adding the probabilities for all paths 
terminating in success. 

(6) Determine the probability of the system failure by adding the probabilities for all paths 
terminating in failure. 
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3.8.4 Example 

An example of an ETA is presented in figure 3-27. The example includes the system and 
scenario being assessed and the resulting event tree. Note that in this example the probability of the 
challenging initiator is assumed to be one and the tree has been pruned to its simplest form by using 
engineering logic. For example, since failure of the float switch is a nonrecoverable failure, its path leads 
directly to a final failure outcome with no alternate paths. In a similar manner since successful operation 
of the pump is an undefeatable success, its path also leads to a final success outcome with no alternate 
paths. 

3.8.5 Advantages 

An ETA provides the following advantages: 

(1) Enables the assessment of multiple, coexisting system faults and failures. 

(2) Functions simultaneously in the failure of success domain. 

(3) End events need not be anticipated. 

(4) Potential single -point failures, areas of system vulnerability, and low-payoff countermea- 
sures are identified and assessed, thereby guiding deployment of resources for improved 
control of risk and optimized utilization of limited resources. 

(5) Failure propagation paths of a system can be identified and traced. This can be a “quick and 
dirty” comparative technique and provides very clear visibility of ineffective countermea- 
sures. 

3.8.6 Limitations 


An ETA possesses the following limitations: 

(1) Address only one initiating challenge. Thus, multiple ETA’s may be needed for a particular 
system. 

(2) The initiating challenge is not disclosed by the analysis, but must be foreseen by the 
analyst. 

(3) Operating pathways must be foreseen by the analyst. 

(4) Although multiple pathways to system failure may be disclosed, the levels of loss 
associated with particular pathways may not be distinguishable without additional analyses. 

(5) Specific, noncomparative success or failure probability estimates are typically difficult to 
find, difficult to achieve agreement on, and to successfully use to drive conclusions. 
Comparative analyses are typically as valuable, with better reception from the program and 
design teams. 


3-54 



Hi|i|i 


-o 

IZ 




BACKGROUND/PROBLEM — A subgrade compartment containing 
important control equipment is protected against flooding by the system 
shown. Rising flood waters close float switch S, powering pumppfrom 
an uninterruptible power supply. A klaxon K is also sounded, alerting 
operators to perform manual bailing, B, should the pump fail. Either 
pumping or bailing will dewater the compartment effectively. Assume 
flooding has commenced, and analyze responses available to the 
dewatering system... 


• Develop an event tree representing system responses. 

• Develop a reliability block diagram for the system. 

• Develop a fault tree for the TOP event Failure to Dewater. 


SIMPLIFYING ASSUMPTIONS: 

• Power is available full time. 

• Treat only the four system components S, P, K, and B. 

• Consider operator error as included within the bailing function, B . 
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Figure 3-27. Example ETA. 
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3.9 Fault Tree, Reliability Block Diagram, and Event Tree Transformations 

3.9.1 Description 

Fault trees (sec. 3.6), RBD’s (sec. 3.5), and event trees (sec. 3.7) are all symbolic logic models. 
Fault trees are generated in the failure domain, reliability diagrams are generated in the success domain, 
and event trees are generated in the success and failure domains. These techniques, described in 
reference 3.13 and presented below, transform any one of the above models into the other two by 
translating equivalent logic from the success to failure or failure to success domain. 

3.9.2 Application 

These techniques are applicable by the analyst that wishes to exploit the benefits of the fault tree, 
RBD, and event tree. Fault trees offer the analyst comprehensive qualitative or quantitative analysis. 
RBD’s offer the analyst a simplistic method to represent system logic. Event trees allow the analyst to 
assess a system in both the success and failure domains. This technique is typically performed in phase 
C but may also be performed in phase B. 

3.9.3 Procedures 


The procedures for transforming a fault tree, RBD, or event tree to either of the other two logic 
models are presented in the following sections. 3 13 

3.9.3. 1 Fault Tree to RBD Transformation 


An RBD represents system component functions that, if these functions prevail, produces 
success in place of a TOP fault event. A fault tree can be transformed into a reliability diagram as 
illustrated in figure 3-28. 

3. 9. 3. 2 RBD and Fault Tree-to-Event Tree Transformation 


An event tree represents path sets in the success branches of the tree and all the cut sets in the 
failure branches of the tree. Therefore, if the path sets and cut sets of a system are known for a certain 
challenge to a system (TOP event of a fault tree), then an event tree can be constructed. 

Cut sets and path sets may be obtained from a reliability diagram as shown in figure 3-29. 

For large complex fault trees, cut sets and path sets are obtainable using the MOCUS algorithm 
described in sections 3. 6. 3. 3 and 3. 6. 3. 4, respectively. 


3-56 














To transform an RBD into an event tree, proceed as shown in figure 3-30. To transform a fault 
tree into an event tree, first transform the fault tree into an RBD (sec. 3.9.3. 1). 


-nm 


Failure of any one of 
these series elements 
makes system failure 
irreversible. 


All of these parallel 
elements must fail to 
produce system failure. 



Figure 3-30. RBD to event tree transformation. 


3. 9. 3. 3 RBD to Fault Tree Transformation 


A fault tree represents system functions which, if they fail, produce TOP event fault in place of 
the success to which the reliability block path lead. The series nodes of an RBD denote an OR gate 
beneath the TOP event of a fault tree. The parallel paths in an RBD denote the AND gate for redundant 
component functions in a fault tree. Therefore, a reliability diagram can be transformed into a fault tree 
as shown in figure 3-31. 

3. 9. 3.4 Event Tree to RBD and Fault Tree Transformation 


An event tree represents path sets in the success branches of the tree and all the cut sets in the 
failure branches of the tree. To transform an event tree into an RBD, reverse the process illustrated in 
figure 3-30. Once the RBD is formed, a fault tree can be formed as illustrated in figure 3-31. Also, an 
event tree can be transformed into a fault tree by inspection as shown in figure 3-32. 

3.9.4 Example 

An RBD and fault tree are transformed from the example event tree presented in figure 3-27, and 
presented in figure 3-33 (a) and (b), respectively. All three of the models represent equivalent logic of the 
system. 
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Figure 3-31. RBD to fault tree transformation. 




*Note that not all events represented here are failures. 


Figure 3-32. Event tree to fault tree transformation. 
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(a) RBD. 



(. b ) Fault tree. 

Figure 3-33. Equivalent logic RBD and fault tree. 
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3.9.5 Advantages 

These techniques allow the analyst to overcome weaknesses of one analysis technique by trans- 
forming a system model into an equivalent logic model as another analysis technique. For example, a 
complex system that may be hard to model as a fault tree might be easily modeled with an RBD. Then, 
the RBD can be transformed into a fault tree, and extensive quantitative or pseudoquantitative analysis 
can be performed. 

3.9.6 Limitations 

These techniques possess the following limitations: 

(1) No new information concerning the system is obtained and the models are only as good as 
the models being transformed. 

(2) The cut sets and path sets required to perform these transformations for large complex 
systems may require many manhours or extensive computer resources to determine. 

3.9.7 Bibliography 
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Altos, New Mexico, 1990. 


3.10 Cause-Consequence Analysis 


3.10.1 Description 

A cause-consequence analysis is a symbolic logic technique described in references 3.6 and 3.14, 
and presented below. This technique explores system responses to an initiating “challenge” and enables 
assessment of the probabilities of unfavorable outcomes at each of a number of mutually exclusive loss 
levels. The analyst starts with an initiating event and performs a forward (bottom-up) analysis using an 
event tree (sec. 3.8). This technique provides data similar to that available with an event tree; however, it 
affords two advantages over the event tree — time sequencing of events is better portrayed, and discrete, 
staged levels of outcome are analyzed. 

The cause portion of this technique is a system challenge that may represent either a desired or 
undesired event or condition. The cause may be a fault tree TOP event and is normally, but not always, 
quantified as to probability. The consequence portion of this technique yields a display of potential out- 
comes representing incremental levels of success or failure. Each increment has an associated level of 
assumed or calculated probability, based on variations of response available within the system. 

A conceptual illustration of how a cause is assessed to understand its consequences is presented 
in figure 3-34. Note that the cause has an associated probability, and each consequence has an associated 
severity and probability. 
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Figure 3-34. Relationship between cause and consequence. 


3.10.2 Application 

This technique is typically applied in phase C or E but may also be applied in phase D. The 
cause-consequence analysis is particularly useful in analyzing command- start/command-stop protective 
devices, emergency response systems, and engineered safety features. Cause-consequence analyses are 
useful in evaluating operating procedures, management decision options, and other nonhardware 
systems. Also, it will evaluate the effect/benefit of subtiered/redundant design countermeasures for 
design trades and assessment. This technique may be used in conjunction with an FTA to provide a 
technique sensitivity assessment. This technique may also be used to compliment an FMEA. 


3.10.3 Procedures 


The procedures, as described in references 3.6 and 3.14, for performing a cause-consequence 
analysis are presented below. 

(1) Identify the initiating event that challenges the system. 

(2) Determine the probability, Pq that this event will occur. This probability may be 
determined from an FTA (sec. 3. 6. 3. 2) or assumed. 

(3) Next, trace the possible consequences to the system from the initiating event. At various 
levels the path may branch with two possible outcomes. Construct the consequence 
diagram by asking the following questions: 3 6 

a. What circumstances allow this event to proceed to subsequent events? 

b. What other events may occur under different system operating circumstances? 
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c. What other system elements does this event influence? 

d. What subsequent event could possibly result as an outcome of this event? 

(4) Use the symbols presented in table 3-8 to construct the consequence diagram. 


Table 3-8. Cause-consequence tree construction symbols. 3 14 


Symbol 

Name 

Description 

6 

OR 

Gate 

Gate opens to produce output when any input 
exists. 

0 

AND 

Gate 

Coexistence of all inputs opens gate and produces 
an output. 

6 

Basic 

Event 

An independent initiating event, representing the 
lower resolution limit of the analysis. 


| 1 ■ 1 

Y 1 N 

Event 

1 


Branching 

Operator 

Output is “Yes” if condition is met and "No” if it 
is not met. Branching operator statement may be 
written in either the fault or the success domain. 
The outputs are mutually exclusive, therefore 

P y +P N = 1 • 

< > 

Consequence 

Descriptor 

End event/condition to which analysis leads, with 
the severity level stated. 


(5) The format of the consequence tree is presented in figure 3-35. Note that all paths lead into 
branching operators or consequence descriptors. The branching operator always has one 
input and two output paths (yes and no). The consequence descriptor has one input, no 
outputs, and is a termination point in the diagram. 

(6) For each branching operator, establish the probability, P; , that the event can happen. 
Therefore, P, and (1-P; ) are the probabilities for the yes and no paths from the branch 
operator, respectively. This step is often difficult and subjective due to a scarcity of data. 
Probability bands are often useful to provide an understanding of the analyst's confidence 
in the delineated probabilities. 

(7) Determine the probability of each consequence descriptor, P r; -, by multiplying event 
probabilities along the path that terminates at that consequence descriptor. 

(8) Finally, determine the severity of each consequence descriptor, 5/. 
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Challenge and for Branching 
Operator Y/N outcomes. 


Figure 3-35. Cause-consequence analysis format. 


3.10.4 Example * 

Problem: 

A copying machine uses an electrically heated drum to fix dry ink to copy paper. The drum 
heater is thermostatically controlled. The drum is also equipped with an automatic overheat safety cutoff 
to prevent damage to the copier. The probability of failure is finite for both the drum thermostat and the 
overheat cutoff. Combustibles are often present in the copying room near the machine. Uncontrolled 
drum temperature can rise high enough to ignite them. The room is equipped with an automatic sprinkler 
system initiated by a heat detector. Employees frequent the room and can initiate an emergency response 
alarm in the event of fire. After a delay, a fire brigade responds to extinguish the blaze. 

The cause-consequence analysis for the above problem is presented in figure 3-36. 


3.10.5 Advantages 

Cause-consequence analyses provide the following advantages: 3 14 

(1) The analysis is not limited to a “worst-credible case” consequence for a given failure. 
Therefore, a less conservative, more realistic assessment is possible. 

(2) Enable assessment of multiple, coexisting system faults and failures. 

(3) End events need not be anticipated. 

(4) The time order of events is examined. 


*This example was provided courtesy of Sverdrup Technology, Inc., Tullahoma, Tennessee . 3 1 
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Note that, because the analysis is exhaustive... 
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Figure 3-36. Example cause-consequence analysis. 


(5) Probabilities of unfavorable system operating consequences can be determined for a 
number of discrete, mutually exclusive levels of loss outcome. Therefore, the scale of 
partial successes and failures is discernible. 

(6) Potential single-point failures or successes, areas of system vulnerability, and low-payoff 
countermeasures are identified and assessed, thereby guiding deployment of resources for 
improved control of risk and optimized utilization of limited resources. 


3.10.6 Limitations 


Cause-consequence analyses possess the following limitations: 3 14 

(1) Address only one initiating challenge. Thus, multiple analyses may be needed for a 
particular system. 

(2) The initiating challenge is not disclosed by the analysis, but must be foreseen by the 
analyst. 
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(3) Operating pathways must be foreseen by the analysts. 

(4) The establishment of probabilities is often difficult and controversial. 

(5) Determining the severity on consequences may be subjective and difficult for the analyst to 
defend. 
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3.11 Directed Graphic (Digraph) Matrix Analysis 


3.11.1 Description 

Directed graph (digraph) matrix analysis, as described in reference 3.15, is a technique using 
matrix representation of symbolic logic models to analyze functional system interactions. Fogic models 
are first generated in the success domain, then converted into the failure domain. However, it should be 
noted that models can be directly created in the failure domain, without first creating the model in the 
success domain. 

This technique consists of four phases. First, the analyst determines combinations of systems or 
combinations of subsystems within a single system for thorough assessment. This phase is parallel to 
determining failure propagation paths using an ETA (sec. 3.8). The second phase consists of 
constructing a digraph model in the success domain, then converting this model to a digraph model in 
the failure domain for each failure propagation path. The third phase consists of separating the digraph 
models into independent models, then determining the singleton and doubleton minimal cut-sets of each 
failure propagation path. Finally, the fourth phase consists of an assessment of the minimal cut sets 
relative to probability of occurrence. 


3.11.2 Application 

This technique, according to reference 3.15, can be used independently or as an element of a 
PRA (sec. 3.15). If this technique is used as part of a PRA, then it is performed after the identification of 
failure propagation paths by ETA but before FTA’s are begun. 3 15 This technique is applied to evaluate 
the failure propagation paths involving several systems and their support systems, or within a single 
system involving several system elements (subsystem, component, part, etc.) and is best applied in phase 
B. 
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3.11.3 Procedures 


Presented below is a summary of the detailed procedures found in reference 3.15 for performing 
a digraph matrix analysis. 

(1) Identify the associated group of systems (or associated system elements of a single system) 
to be thoroughly evaluated. Use event trees (sec. 3.8) to identify failure propagation paths. 
For a complete analysis, identify every credible initiator to an undesirable event and 
prepare an event tree that illustrates each specific failure propagation path. 

a. Acquire pertinent information concerning the collection of systems to be assessed, such 
as design specifications and packages, safety assessment reports (such as PHA’s, sec. 
3.2), and prior safety or reliability studies. 

b. Study checklists of potential initiating challenges. From these checklists develop a list 
of initiators that are applicable to the systems being studied. 

c. Develop event trees for each initiating challenge to the system. 

d. Prepare a list of failure propagation paths from step lc. Assume unity probability for all 
systems required to work in the failure propagation path. This simplifying assumption 
leaves only failure propagation paths that are combinations of systems that must fail for 
a serious threat to be posed. 

(2) Construct a digraph model for each possible failure propagation path. Use a backward, top- 
down approach to construct a top-level digraph, then expand each element into its own 
digraph. Continue expanding the elements of new digraphs until the desired resolution level 
of the analysis is reached. An outline of the steps involved in producing the digraphs is 
presented below. 

a. Create a success domain digraph model for each success path. Connect upstream 
elements to a downstream element with an AND gate if the upstream element relies on 
the successful operation of all the downstream components. Connect upstream elements 
to a downstream element with an OR gate if the upstream element relies on the 
successful operation of only one of two or more downstream elements. The symbols for 
AND and OR gates for a digraph are different than those used for a fault tree, however 
they represent the same logic as the fault tree symbols. A comparison between the 
digraph and fault tree symbols is presented in figure 3-37. 

b. Form a failure domain model by taking the model generated in step 2a and interchange 
all AND gates with OR gates and all OR gates with AND gates. This failure domain 
model represents a path for failure propagation. 

c. Form an adjacency matrix that represents the digraph. The matrix is constructed by the 
process illustrated in figure 3-38. 
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Event C will occur only if 
event A or event B occurs. 


Figure 3-37. Comparison between digraph and fault tree logic gates. 


d. Next link all connected elements in the adjacency matrix. This is accomplished by 
processing the adjacency matrix with the reachability code. This code is described in 
detail in reference 3.15. The output of this code will show all elements connected by a 
path and illustrate which elements can be reached from a specific element, therefore all 
possible paths between pairs of nodes in the network. Next, use this information to 
determine singleton and doubleton cut sets. 

e. Determine minimal singleton and doubleton cut sets from the cut sets determined in 
step 2d. 

(3) Subdivide the digraph into independent digraphs if the success domain digraph model 
becomes too large to determine singleton and doubleton cut sets for the computer platform 
being used. Then determine singleton and doubleton minimal cut sets of the smaller 
independent digraphs. 

(4) Assess the singleton and doubleton minimal cut sets. This assessment can be conducted in a 
manner similar to that for a conventional PRA (sec. 3.15) in which risk is assessed with the 
probability of the cut sets occurring and the severity of the consequence of the failure 
propagation path. 
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Figure 3-38. Construction of digraph adjacency matrix. 


3.11.4 Example 

An example digraph matrix analysis, adapted from reference 3.15, for a simple system is 
illustrated in figure 3-39. The system consists of two redundant power supplies to power a motor that 
drives a pump. The success domain model of this system is presented in figure 3-39(a). Note that this 
model represents the success path for successful operation of the pump. The failure domain model, 
presented in figure 3-39 (b), was generated by replacing the OR gate in the success domain model with 
an AND gate. Inspection of the two models suggests that for simple systems the failure domain model 
can easily be generated without first generating the success model. In cases with more complex systems, 
first generating a success domain model may prove to be beneficial. 

The adjacency matrix and adjacency elements are presented in figures 3-39(c) and (d), 
respectively. The adjacency matrix illustrates whether there is a direct path from node i to node j. If 
matrix element (ij) = 1, there is a path from node i to node j. For example, element (M.P) = 1, which 
means there is a straight (uninterrupted) and unconditional path between the motor and pump. If element 
(ij) = 0, there is no path from node i to j. For example, element (PS- 1, PS-2) = 0, which means there is 
a no straight path between the main power supply and the auxiliary power supply. If the adjacency 
element (ij) is ^ 0 or 1, then there is a second component that must fail along with component i to cause 
component j to fail. For example, adjacency element (PS-1, M) is equal to PS-2 (nonzero or 1 value). 
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This symbol represent the second component that must fail, given the failure of PS- 1, to cause M to fail 
to operate (i.e., failure of both the main and auxiliary power supplies will cause the motor not to 
operate). 

The reachability matrix and reachability elements are presented in figure 3-39(<?) and (/), respec- 
tively. The methodology to generate the reachability matrix from the adjacency matrix is presented in 
reference 3.15. Simply stated, the reachability matrix illustrates the pairs of nodes that a path exists 
between, by connecting linked pairs from the adjacency matrix. Therefore the reachability matrix 
illustrates the complete pathways (through linked node pairs) of the graphical model elements illustrated 
by the adjacency matrix. Processing the adjacency matrix into the reachability matrix yields the paths 
between all pairs of nodes. The reachability elements are derived from the reachability matrix in the 
same manner that adjacency elements are derived from the adjacency matrix. Note, in this example, that 
the reachability elements include all the adjacent elements and the new information that if both PS - 1 and 
PS - 2 fail, then P will not operate (even though neither PS - 1 or PS - 2 are directly adjacent to P). 
Therefore, the reachability matrix yielded the new information that if both power supplies failed, the 
pump will not operate. 

The summary matrix presented in figure 3-39(g) illustrates which components can lead to failure 
of the pump, P. If an is entered as a matrix element (/,/') and either i or j is a value of 1, then the other 
corresponding component i or j is a singleton. The only singleton in this system is the motor, i.e., the 
single failure of the motor will cause the pump not to operate. If a is entered as a matrix element QJ) 
that corresponds to component i and component j, then component i and component j form a doubleton. 
The only doubleton of this system is the pair of redundant power supplies, i.e., failure of both the main 
and auxiliary power supplies will cause the pump not to operate. 

Obviously, in this example the singletons (single point failures) and doubletons (double point 
failures) could have easily been identified without performing a digraph matrix analysis. However, for 
complex systems which are modeled with many nodes and logic gates, this technique allows 
determination of singletons and doubletons which otherwise would not be as readily identified. 


3.11.5 Advantages 

The digraph matrix analysis provides the following advantages: 315 

(1) The analysis allows the analyst to examine each failure propagation path through several 
systems and their support systems in one single model. Unlike the FTA with failure propa- 
gation paths divided in accordance to arbitrarily defined systems, this approach allows 
more rigorous subdividing of the independent subgraphs. 

(2) Since the technique identifies singleton and doubleton minimal cut sets without first deter- 
mining all minimal cut sets, considerable computer resources can be saved over other 
methods such as the FTA. 
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(b) Failure domain model. 
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(c) Adjacency matrix. 


(d) Adjacency elements. 
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Figure 3-39. Example digraph matrix analysis — Continued 
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(g) Summary matrix. 

Figure 3-39. Example digraph matrix analysis — Continued. 


3.11.6 Limitations 

Digraph matrix analyses possess the following limitations. 3 15 

(1) Trained analysts and computer codes to perform this technique may be limited. 

(2) For particular types of logic models, complete treatment may require more computer 
resources than FTA’s. 

3.11.7 Bibliography 

Grumman Space Station Division, “Digraph Analysis Assessment Report.” Reston Virginia, October 
1991. 

Kandel, A., and Avni, E.: “Engineering Risk and Hazard Assessment.” vol. 2, CRC Press Inc., Boca 
Raton, Florida. 


3.12 Combinatorial Failure Probability Analysis Using Subjective Information 

3.12.1 Description 

The combinatorial failure probability analysis using subjective information is described in refer- 
ence 3.16 and presented below. This technique was developed by the System Effectiveness and Safety 
Technical Committee (SESTC) of the American Institute of Aeronautics and Astronomies (AIAA), in 
1982. This technique provides the analyst a procedure to propagate probability data derived from the 
subjective probability scales defined in MIL-STD-882C. 3 2 
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3.12.2 Application 


This technique is typically performed in phase C and is applicable when no quantitative failure 
probability data are available and may be used in conjunction with other analyses such as an RBD (sec. 
3.5), FTA (sec. 3.6), STA (sec. 3.7), ETA (sec. 3.8), and cause-consequence analysis (sec. 3.10). 


3.12.3 Procedures 


The procedures, as described in reference 3.16, for a combinatorial failure probability analysis 
using subjective information are presented below. 

(1) Arbitrary, dimensionless “probability values” have been assigned to the probability incre- 
ments (frequent, probable, occasional, remote, and improbable) defined in MIL-STD- 
882C. 3 2 The subjective scale for these arbitrary values is presented in table 3-9. Descriptive 
words and definitions for the level of the scale are also given in this table. 


Table 3-9. Combinatorial failure probability analysis subjective scale. 


AIAA/SESTC 

MIL-STD-882C 

Threshold 

Levels 

Probability 

Level 

Level 

Descriptive Word 

Definition 

8xl(T 2 to 
1.00000 

3X1CT 1 

A 

Frequent 

Likely to occur frequently. 

8x1 0~ 3 to 
8x1 0~ 2 

3X1CT 2 

B 

Probable 

Will occur several times in life of an item. 

8xl0“Ao 

8xicr 3 

3x1 cr 3 

C 

Occasional 

Likely to occur sometime in life on an item. 

8xl0 -5 to 
8X10 -4 

3X10 -4 

D 

Remote 

Unlikely but possible to occur in life of an 
item. 

0.00000 to 
8xl<r 5 

3X10- 4 

E 

Improbable 

So unlikely if can be assumed occurrence may 
not be experienced. 


H 4 . . 

Arbitrarily selected, dimensionless numbers. 

Table provided courtesy of Sverdrup Technology, Inc., Tullahoma, Tennessee. 

(2) Estimate subjective failure probabilities of contributor events or conditions using the scale 
defined in MIL-STD-882C. 3 2 Select and consistently apply the same probability exposure 
interval (operating duration or number of events) for every initiator probability estimate 
used in the analysis. 

(3) Correlate the subjective estimate (step 2) with the arbitrary, dimensionless values (step 1). 
Propagate these values in the same manner as quantitative data is combined in classical 
numerical methods (such as presented in figs. 3-18 and 3-19). 

(4) Convert the final probability number resulting from propagation (step 3) back into the sub- 
jective scale defined in MIL-STD-882C. 3 2 
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3.12.4 Example 

The following example* uses this subjective combinatorial technique in a fault tree problem. 
Problem/B ackground: 

• A large rotating machine has six main-shaft bearings. Replacement of a bearing costs $18,000 
and requires 3 wk of down time. 

• Each bearing is served by: 

• pressurized lubrication oil 

• a water-cooled jacket 

• a temperature sensing/alarm/shutdown system. 

• In addition, there are sensing/alarm/shutdown systems for: 

• lube pressure failure 

• cooling water loss of flow. 

• If they function properly, these systems will stop operation of the rotating machine early 
enough to prevent bearing damage. (System sensitivity makes the necessary allowance for 
machine “roll-out” or “coasting.”) 

• Failure records for the individual system components are not available, but probabilities can 
be estimated using the subjective scale of MIL-STD-882C. 3 2 

What is the probability that any one of the six bearings will suffer burnout during the coming 
decade? 

The system schematic and fault tree are presented in figure 3-40 (a) and (b), respectively. Note 
both the arbitrary subjective probability value and letter representing the relevant probability level from 
table 3-9 are presented for each fault tree initiator. 


3.12.5 Advantages 

This technique allows the analyst to perform a probabilistic assessment based on the exercise of 
subjective engineering judgment when no quantitative probability estimates are available. 


3.12.6 Limitations 


This technique should only be used when actual quantitative failure rate data is not available. The 
use of actual quantitative data is preferred over this method. This tool should only be used for 
comparative analysis only. Data and results, unless used in a comparative fashion, may be poorly 
received. 


*This example was provided courtesy of Sverdrup Technology, Inc., Tullahoma, Tennessee . 316 
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s. 


Bearing Burnout Loss Penalty: 

• $18,000 Replacement Costs 

• 3-Week Interruption of Use 

(a) System schematic. 



0 b ) System fault tree. 

Figure 3-40. Example combinatorial failure probability analysis. 
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3.13 Failure Mode Information Propagation Modeling 


3.13.1 Description 

Failure mode information propagation modeling is a qualitative analysis method described in 
reference 3.17 and presented below. This technique involves separating a system into its basic functional 
components and examines the benefit of measuring precedent failure information that may be 
transmitted between components of a system. This information may be transmitted during the initial 
outset of a variety of failure modes. The technique provides insight into both the types of information 
that should be measured to safeguard the system, and location within the system at which sensors might 
be appropriately positioned. 


3.13.2 Application 

This technique effectively directs resource deployment to optimally safeguard a system against 
potential failures by identifying measurement requirements. These requirements are defined in terms of 
measured parameter, sensor type, and sensor location. This technique is best applied in phase C but may 
also be applied in phase D. 


3.13.3 Procedures 


The procedures, as described in reference 3.17, to perform failure mode information propagation 
modeling are presented below. 

(1) Divide the system into its principle functional components and assign a number to each 
component. Like the FMEA (sec. 3.4), the resolution of this analysis is dependent upon the 
level (i.e., subsystems, assemblies, subassemblies, or piece parts) to which the system 
elements are resolved. 

(2) Identify the physical links (energy flow and shared stress) between the components of the 
system. These links include such items as electrical power, air flow, liquid flow, gas flow, 
thermal heat transfer, friction, spring, rolling element, etc. 

(3) Identify and record the failure modes for each component and assign a letter to each failure 
mode for each component. 

(4) Identify and record the flow of failure mode information at each physical link that is 
available externally to each component and transmitted to one or more other components. 

(5) Classify the failure mode information constituents by their signal characteristics (e.g., 
thermal, pressure, acceleration, etc.). 

(6) Identify the minimal success sets of the sensor network. A minimal success set is a sensor 
group that encompasses all failure modes. 

(7) Assess the various minimal success sets in terms of feasibility, cost, and effectiveness. The 
following questions should be asked: 

a. Feasibility. Do the sensors currently exist or can they be developed? Can they be 
obtained in time to satisfy schedule requirements? 
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b. Cost. Is the cost of installing, maintaining, and operating the sensor network less than 
the cost of the failure that the system is being safeguarded against? 

c. Effectiveness. Are there other preventive maintenance activities more effective than 
installing a sensor network? Will the sensing network forewarn before the start of 
system failures or does it just announce system crashes? Will the sensors impede 
normal system operation? Will they degrade system performance? Will they pose any 
new hazards to the system? Will the sensor network operate dependably? Will the 
sensors have adequate sensor redundancy? 


3.13.4 Example 

The following example* uses failure mode information propagation modeling to a sensor 
network success set for a system. 

Problem: 

Consider a ventilating fan powered by an electric motor through a belt drive. A common frame 
structure supports both the motor and a bearing, through which power is delivered to the fan. (Consider 
motor bearings as integral parts of the motor.) Assume a constant aerodynamic fan load. A schematic of 
the system is presented in figure 3-41 (a). Determine sensor network minimal success sets for the system. 

Solution: 

(1) Perform steps 1-5 identified in section 3.13.3. These steps are explained below and illus- 
trated in figure 3-4 1(b). 

a. Step 1. Divide the system into its principle functional components and assign a number 
to each component. These are the electrical motor, fan belt, fan, frame, and bearing. 

b. Step 2. Identify the physical links (energy flow and shared stress) between the compo- 
nents of the system. The electric motor, for example, has electrical power input, is 
linked to the fan belt by friction, and is mechanically and thermally linked to the frame. 

c. Step 3. Identify and record the failure modes for each component and assign a letter to 
each failure mode. For example, the failure modes for the fan include shaft or rotor 
binding, bearing vibration, open winding, and shorted winding. 

d. Step 4. Catalog the flow of failure mode information at each physical link that is 
available externally to each component and transmitted to one or more other 
components. For example, for the mechanical link between the electric motor and 
frame, the failure information available includes electric motor bearing vibrations (1- 
B), fan belt slipping and breaking (2-A/B), and bearing binding (5-A). 

e. Step 5. Classify the failure mode information constituents by their signal 
characteristics. For example, the electric motor bearing vibration (1-B) and fan bearing 
vibration (5-B) can be monitored by an accelerometer at test point 4/1 (between frame, 
component 1 and electric motor, component 4). 


*This example was provided courtesy of Sverdrup Technology, Inc., Tullahoma, Tennessee. 3 16 
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(2) From the information displayed in figure 3-4 1(^), construct a matrix of failure mode versus 
sensor type (with each test point identified). Determine the minimum success sets of mea- 
surement sensors. These sets are sensor groups that encompass all failure modes. The 
matrix and minimum success sets for this system are presented in figure 3-4 1(c). 

3.13.5 Advantages 

Information propagation modeling provides the following advantages: 317 

(1) Allows the analyst to identify measurement requirements, that, if implemented, can help 
safeguard a system by providing warnings at the onset of a failure mode that threatens the 
system. 

(2) Compliments an FMEA. 

3.13.6 Limitations 

Information propagation modeling possesses the following limitations: 3 17 

(1) This technique is only applicable if the system is operating in a near-normal range, and for 
the instant of time immediately prior to the initiation of a failure mode. 

(2) Externally induced and common cause faults are not identified or addressed. 

(3) The risks of the failure modes are not quantified in terms of criticality and severity. 

(4) The propagation of a failure through the system is not addressed. 


3.14 Probabilistic Design Analysis 


3.14.1 Description 

A PDA, as described in references 3.8 and 3.18, is a methodology to assess relative component 
reliability for given failure modes. The component is characterized by a pair of transfer functions that 
represent the load (stress, or burden) that the component is placed under by a given failure mode, and 
capability (strength) the component has to withstand failure in that mode. The variables of these transfer 
functions are represented by probability density functions. Given that the probability distributions for 
both the load and capability functions are independent, the interference area of these two probability 
distributions is indicative of failure. Under these conditions, a point estimate for failure of the 
component relative to the failure mode under consideration can be determined. 


3.14.2 Application 

A PDA can be used to analyze the reliability of a component during phase C of a program. The 
PDA approach offers an alternative to the more traditional approach of using safety factors and margins 
to ensure component reliability. This traditional approach is vulnerable if significant experience and 
historical data are not available for components similar to that which is being considered. 3 8 3 18 
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Elements: 

• Electric Motor 

• Fan Belt 



(a) System schematic. 



Model. 


(b) 


Figure 3-41. Example failure mode information propagation model — Continued 
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Minimal Success Sets* 




*Sensor groups that envelope all 
failure modes 


(c) Minimal success sets. 

Figure 3-41. Example failure mode information propagation model — Continued. 


A 


3.14.3 Procedures 


The procedures, adapted from reference 3.8 and 3.18, for performing a PDA in the context of a 
total design reliability program for a system are presented below. 

(1) Specify the system design requirements. These requirements should be stated in clear and 
concise terms that are measurable and verifiable. 

(2) Identify variables and parameters that are related to the design. 

(3) Identify the failure modes of the system by using a method such as a FMEA (sec. 3.4). 

(4) Confirm the selection of critical design parameters. 

(5) Establish relationships between the critical parameters and organizational, programmatic, 
and established failure criteria. 

( 6 ) Ascertain the reliability associated with each critical failure mode with the following proba- 
bilistic analysis method: 

a. Identify the random variables that effect the variation in the load to be imposed on the 
component for the given failure mode. Incorporate these random variables into a 
transfer function that represents this load (stress, or burden). 

Load Transfer Function: L = // j (X], X 2 , A 3 , 
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b. Identify the random variables that affect the variation in the capability of the component 
to withstand the load imposed for the given failure mode. Incorporate these random 
variables into a transfer function that represent this capability (strength). 

Capability Transfer Function: C = gc(Y\, Y 2 , Y 3 , ....Y m ). 

c. Gather data to perform the load and capability calculations. 

d. Determine probability distributions of the load (stress, or burden) and capability 
(strength) of the failure mode. Consider each variable of the transfer function as a prob- 
ability density function (illustrated in figure 3-42). The density function can be repre- 
sented as either a discrete variable distribution using empirical test data, or as a continu- 
ously variable form of the density function. 

Note: The area under an entire probability density function curve is equal to a 
probability of one, therefore a range between two values of the independent random 
variable of a density function curve is equal to a probability less than or equal to one. 
Probability density functions of both load and capability continuous random variables 
for a given failure mode are presented in figure 3-43. Also illustrated in this figure is 
the interference of the load and capability density functions. For independent load and 
capability functions, this interference is indicative the failure mode will occur. In figure 
3-43, both density functions are normal distributions with different means and 
variances. However, generally one or both of these density functions may be an 
exponential, log normal, gamma, Weibull, or other distribution. 

e. Calculate the reliability ( R ) for the failure mode from the load and capability 
distributions. Reliability is the probability that the failure mode will not occur. The 
expression for reliability is: 

R = \-Pp. 


The expression for Pp, is dependent upon the type of load and capability distributions. 
Expressions for Pp for various distributions are found in most advanced statistics text- 
books and handbooks. Expressions for Pp between combinations of exponential, log 
normal, gamma, and Weibull distributions are found in reference 3.8. 

(7) Assess the reliability for each critical failure mode, including load and capability in this 
assessment, then modify the design to increase reliability. Repeat the process until the 
design reliability goals or requirements are met. 

(8) Perform trade studies (sec. 2.1) to reassess and optimize the design for performance, cost, 
environmental issues, maintainability, etc. 

(9) Repeat step 8 for each critical component for the system. 

(10) Determine the relative reliability of the system. 

(1 1) Repeat the above steps to optimize system reliability. 
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Figure 3-43. Interference between load and capability density functions. 3 - 8 


3.14.4 Advantages 

A PDA provides the following advantages: 

(1) Allows the analyst a practical method of quantitatively and statistically analyzing the 
relative reliability of a system during the design phase. 3 8 Therefore PDA’s can be used to 
determine valuable areas of the design and aid in determining the resource allocation during 
the test and evaluation phase. 

(2) This technique mandates that the analyst address and quantify the uncertainty of design 
variables and understand its impact on system reliability of the design. 3 8 

(3) The PDA approach offers a more accurate and truly quantitative alternative method to the 
more traditional approach of using safety factors and margins to ensure component 
reliability. 3 - 8 318 

(4) The technique provides a more precise method for determining failure probabilities to 
support FTA’s than does use of subjective methods. 


3.14.5 Limitations 


A PDA possesses the following limitations: 

(1) The analyst must have experience in probability and statistical methods to apply this 
technique. 3 8 

(2) Determining the density functions of the random variables in the load and capability 
transfer functions may be difficult. 3 18 

(3) Historical population data used must be very close to the as-planned design population to 
be viable. Extrapolation between populations can render the technique nonviable. 
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(4) This technique identifies the relative probabilities that various failure modes will occur, but 
does not address the severity of the failure modes. Therefore, this technique should be used 
as one element among other elements of a PRA (sec. 3.15) to assess the risk associated with 
the various failure modes. 


3.15 Probabilistic Risk Assessment 


3.15.1 Description 

A PRA is a general term given to methodologies that assess risk. Although PRA methods are 
customarily thought of as being quantitative, these methods can be either subjective (as by use of the risk 
assessment matrix, sec. 3.1), or quantitative in nature. 

According to reference 3.6, a PRA generally consists of three phases. During phase 1, the system 
is defined, hazards are identified, elements of the system vulnerable to hazards are identified, and the 
overall scope of types of hazards to be assessed is defined. PHA’s (sec. 3.2), are typically performed 
during phase 1. 

During phase 2, the failure propagation paths and probabilities are established. ETA (sec. 3.8), 
FTA (sec. 3.6), FMECA (sec. 3.4) and/or cause-consequence analysis (sec. 3.10) are performed. 

Finally, during phase 3, a consequence analysis is performed. Severity is established. Then, an 
assessment of risk is performed in terms of probability and severity, and by comparison to other societal 
risks. 


3.15.2 Application 

A PRA is performed to identify consequence of failure in terms of potential injury to people, 
damage to equipment or facilities, or loss of mission requirements. The PRA is typically performed in 
phase C. 

3.15.3 Procedures 


The following procedures, adopted from reference 3.6, offer guidance in performing a 
probabilistic risk assessment:. 

(1) Phase 1 (activities performed during the preliminary design stage). 

a. Define the system to be assessed, identify the elements (targets) of the systems that are 
susceptible to hazards, and from an overall perspective identify potential hazards. 

b. Perform a PHA (sec. 3.2). In performing a PHA, the analyst: (1) identifies targets, (2) 
defines the scope of the system, (3) recognizes the acceptable risk limits, (4) identifies 
hazards, (5) assesses the risk for each hazard and target combination in terms of proba- 
bility and severity, (6) and if the risk are unacceptable determines countermeasures to 
mitigate the risk, and (7) and repeats the assessment with the countermeasures 
incorporated. 
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(2) Phase 2 (activities initiated after accomplishing hardware and configuration selections). 

a. Identify failure propagation paths with techniques such as an ETA (sec. 3.8). In 
performing an ETA, the analyst (1) identifies an initiating challenge to the system, and 
(2) determines the alternate logic paths from the initiating event. 

b. Determine initiators and propagate probability of failure with methods such as FTA 
(sec. 3.6). Probability of failure modes can also be determined with the probabilistic 
analysis method presented in section 3.14. 

c. A cause-consequence analysis (sec. 3.10) may be performed to establish both failure 
propagation path and probabilities of causes and consequences. 

d. A digraph-matrix analysis (sec. 3.11) may be performed after the ETA is complete and 
before FTA’s have begun. 315 

e. An FMECA (sec. 3.4) may be performed. Examine all failure modes and criticality 
ranking of each system element. 

(3) Phase 3 (perform a consequence analysis). 

a. Establish the severity of the failure modes. 

b. Assess risk of all failure modes in terms of severity and probability. 

c. Calibrate the risk of the system being examined by comparing it to other known societal 
risks. 


3.15.4 Advantages 

Assessing risk avoids unknowingly accepting intolerable and senseless risk, allows operating 
decisions to be made, and improves resource distribution for control of loss resources. 3 1 


3.15.5 Limitations 


A PRA possesses the following limitations: 

(1) Probabilistic risk assessment requires skilled analysts. If the analyst is untrained in the 
various tools required, the tool could be misapplied or the results misinterpreted. 

(2) Depending on the size and complexity of the system being assessed, significant manhour 
and/or computer resources may be needed to complete. 

(3) Sufficient information and data may not be available to perform a thorough assessment. 
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4. DESIGN-RELATED ANALYTICAL TOOLS 


Two design-related analytical tools (sensitivity analysis and tolerance stackup analysis) that can 
be useful to systems engineering are discussed in this section. In addition, Geometric Dimensioning and 
Tolerancing, ANSI-Y-14.5, is discussed. This section is included to give the systems engineer an under- 
standing of the standard methods of dimensioning and tolerancing. 

A summary of the advantages and limitations of each tool or methodology discussed in this 
section is presented in table 4-1. 


4.1 Sensitivity (Parametric) Analysis 


4.1.1 Description 

In sensitivity analysis, sensitivity functions (or coefficients of influence) are generated by taking 
partial derivatives with respect to each parameter that affects the outcome of a relationship. 

4.1.2 Application 

Sensitivity analysis typically should be performed in phase C or D. This analysis can be used for 
nearly any type of relationship. Sensitivity analysis is especially useful when environmental conditions 
can change, when factors such as age affect performance, or when manufacturing tolerances affect 
performance. Sensitivity analysis can show which parameters affect a system the most or least. This can 
facilitate optimizing a system, reducing variability, or adjusting a system for wear or changing 
conditions. Typical examples of the use of sensitivity analysis are manufacturing formulation and 
processes (e.g., bond strength, burn rate, erosion rate, or material strength). 

4.1.3 Procedures 


The procedure for obtaining the sensitivity of a relationship by analytical methods is as follows: 

(1) Generate an equation for the relationship under consideration. 

(2) Find the coefficients of influence 4 1 by taking the partial derivatives for each parameter 
under consideration. 

(3) Solve the equations for the coefficients of influence to find the sensitivity at given 
conditions. 

An alternate approach to approximate sensitivity is to assume a straight-line relationship between 
two points in the sample space of the relationship, and to solve the relationship for two conditions repre- 
sented by two values of the parameters in question. This method is often preferred for relationships with 
parameters that are interrelated, such as throat area and exit pressure in the thrust equation. 
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a smallest unit of measurement greater than a 
0.0005 tolerance. 

(2) It may be superfluous to combine tolerances from 
different manufacturing processes, e.g., 
machining and casting. 






















4.1.4 Example 


In the following hypothetical example, the sensitivity of pressure with respect to throat area is 
being determined. The equation for this analysis is the pressure equation. The equation for pressure is: 


r b 


C p A 
g A 


(4.1) 


where P c is the chamber pressure, r/ 7 is the propellant burn rate, C* is the propellant gas characteristic 
exhaust velocity, p is the propellant density, g is gravity, A s is the propellant burn surface area, and A* is 
the throat area. To find the sensitivity of pressure to motor throat area, take the partial derivative of 
equation (4.1) with respect to A*. A P c is taken over a narrow range where r/ 7 is approximately constant. 
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(4.2) 


where d designates a partial derivative. The sensitivity is found by substituting values for the variability 
into the partial derivative equation. Numbers can be substituted into equation (4.2) to obtain the slope at 
a particular value of A*. It is intuitively obvious that the relationship between the partial derivative and 
A* is both negative and inversely proportional to A* 2 . 

Another example of the approximation method is the substitution of selected values into the 
thrust equation (4.6). The sensitivity of thrust to throat area is to be investigated for a hypothetical motor 
with the following characteristics: 


A s = 300 in 2 
A* = 1.9 in 2 , 2.1 in 2 
A e = 10 in 2 

p = 0.06 lbm/in 3 
y = 1.2 
r b = 0.5 in/s 
C* = 5100 in/s 
g = 386.40 in/s 2 . 

The first step is to calculate the chamber pressure, substituting into equation (4.1), using the first 
value of A* which is 1.9 in 2 . The next step is to calculate Mach number ( M ) iteratively from equation (4.3): 



\2 


U ) 


AT 


_Y +1 


1+ - — - M~ 


- A +1 

Y -1 


(4.3) 
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The third step is to calculate nozzle exit plane pressure ( P e ) from equation (4.4). 


Pe. 

Pc 


1 



Y ~ 1 
2 


M 


Y 

Y— 1 


(4.4) 


The next step is to calculate the thrust coefficient ( Cf) from equation (4.5). 
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(4.5) 


The final step is to calculate thrust (7) from equation (4.6). 

T = C F A* P c . (4.6) 

The above calculations should be performed again, using A* = 2.1 in 2 . The values obtained from 
both calculations are shown in table 4-2. 


Table 4-2. Sensitivity analysis calculations. 



Pc 

M 

Pe 

Cf 

T 

A* = 1.9 in 2 

62.52 

2.82 

1.87 

1.50 

177.62 

A* = 2.1 in 2 

56.57 

2.75 

1.93 

1.47 

174.60 


Conclusion: 

The thrust ( T ) is inversely proportional to the throat area (A*). 


4.1.5 Advantages 

The effect of each parameter can be assessed to determine which parameters have the greatest 
effect on the outcome of a process, and which parameters can yield the most benefit for adjustment. 

4.1.6 Limitations 


It is often not easy to isolate a variable to obtain a second derivative. For example, when 
obtaining the sensitivity of thrust to throat diameter, changing a throat diameter not only changes motor 
pressure, but changes the nozzle expansion ratio and exit pressure. The pressure ratio is typically found 
by iteration or by tables. If the approximation approach above is taken, care must be used to ensure a 
small enough range for the parameter values to achieve the desired accuracy. 
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4.2 Standard Dimensioning and Tolerancing 


4.2.1 Description 

Dimensioning and tolerancing on drawings is complicated enough to yield confusion, unless 
standardized methods are employed at all stages of a project life from design to manufacture. Standard 
dimensioning and tolerancing per ANSI-Y-14.5 is an internationally recognized method of stating 
dimensions and tolerances. 

4.2.2 Application 

Standard dimensioning and tolerancing is typically applied in phase C but the technique could 
also be applied in phase D. Standard dimensioning and tolerancing allows the design engineer to 
indicate how tolerances are to be applied. This information is understood by draftsmen, manufacturing 
engineers, and machinists to assure the form, fit, and function intended by the design engineer (or 
systems engineer). Some of the methods of specifying dimensions and tolerances are discussed here. 


4.2.3 Procedures 

This section explains how dimensions and tolerances are specified on design drawings. 4 3 
Following is a list of feature controls used to specify how a tolerance is to be applied to a design feature, 
from ANSI-Y-14.5: 


— 

straightness 


angularity 

n 

flatness 


perpendicularity 

O 

circularity 

// 

parallelism 

/a 

cylindricity 


position 


profile of a line 

© 

concentricity 


profile of a surface 

/ 

circular runout 

-B- 

datum identifying 
letter 

4L 

total runout 

XXX 

basic dimension 

(XXX) 

reference dimension 


A basic dimension is contained in a box (unless otherwise specified on the drawing). Basic 
dimensions are the controlling dimensions on a drawing, and have no tolerances associated with them. 
Basic dimensions set up a dimensional pattern, such as a bolt pattern. The locations of the features in the 
pattern (e.g., bolt holes or threads) are toleranced using true-position tolerances. Often the title block of a 
drawing will indicate standard tolerances peculiar to that drawing that will apply to all basic dimensions 
shown without a specified tolerance. A tolerance is shown for each significant digit used in the basic 
dimensions on the drawing. For example, a tolerance of + 0.1 may apply to all basic dimensions with 
one significant digit. 
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Reference dimensions are the result of basic dimensions. In the example below, an inner and 
coincident outer diameter are specified; the thickness is a reference dimension. In this situation, the inner 
and outer diameters are of primary importance; the thickness is of secondary importance. 

A rectangular box is used as a feature control box. The symbol in the first section of the box is 
for the type of tolerance (e.g., true position). The first symbol in the second section is the type of 
measurement (a diametrical tolerance is shown in fig. 4-1). The number is the size of the tolerance. The 
second symbol in the second section (a circle with the letter “M,” “L,” or “R”) specifies the relation of 
the tolerance with the size of the feature. The third (and any subsequent) section specifies which data are 
used (which feature or dimension the tolerances concern). 

4.2.4 Example 

Following is a hypothetical fixed nozzle assembly used to show the purpose of dimension and 
tolerance methods: 



Figure 4-1. Example of dimensioning and tolerancing. 


In this example, datum A is defined by the throat of the nozzle, thus datum A is the axis of the 
throat. The nozzle exit is referenced to datum A. The true position of the exit is to be within ±0.030 of the 
throat axis (datum A), and the exit plane is to be within a 0.020 tolerance zone perpendicular to the throat 
axis. The true position tolerance is not affected by the feature size of the throat diameter. (The “R” inside 
the circle indicates that the position tolerance is applied “regardless of feature size.” An “M” inside the 
circle would denote that the position tolerance applies to “maximum material condition;” thus the 
tolerance can be relaxed by an amount commensurate with the difference that the size of the feature is 
less than the maximum allowable size. An “F” inside the circle would denote “least material condition” 
where the tolerance applies to the smallest feature size allowable.) The exit plane also defines datum B. 

The boss at the end of the nozzle is controlled by a total runout tolerance. The surface is to be 
within a 0.010 tolerance zone perpendicular to the axis made by the throat and exit (datums A and B). 

The threads of the nozzle are to be concentric to the throat and exit axis within 0.005, and the axis of the 
threads is to be within ±0.015 of the throat axis. Note that for the profile type tolerance controls (e.g., 
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runout or perpendicularity), the number defines a tolerance zone. This means that the total “width” of the 
acceptable deviation is defined by the tolerance. Thus a tolerance zone of 0.020 is analogous to a ±0.010 
tolerance. For position tolerances, the number call out is ±, thus the axis of the nozzle exit must fall 
inside a circle of 0.030 radius around the throat axis. 

Note that the tolerances in this example control the thrust vector. The length of the nozzle is con- 
trolled by a basic dimension. The exit is true-positioned to the basic dimension from the nozzle throat, 
and the required perpendicularity to the throat axis is greater than the true position tolerance. The nozzle 
exit is toleranced to keep the thrust vector in line (within a certain amount) with the throat axis. The 
nozzle boss is controlled by runout to the axis defined by the throat and exit plane. The boss surface 
tolerance is to facilitate a consistent seal with the motor. The thread is controlled by concentricity to the 
same axis to keep the thrust axis in line with the motor axis. It can be seen that the thickness of the boss 
is not a controlling dimension; it is a reference dimension. If this dimension were not specified, the form, 
fit, or function of the component would not be affected. 

4.2.5 Advantages 

Dimensioning and tolerancing per ANSI-Y-14.5 is fairly standard. In addition, some aspects of 
dimensioning and tolerancing per ANSI-Y-14.5 are better suited for production. For example, true posi- 
tioning allows for a circular tolerance zone, whereas putting tolerances to rectangular coordinates allows 
a square tolerance zone. Thus, a functional part that would comply with true position tolerances may not 
comply with rectangular tolerances. Dimensioning strategy can minimize the cumulative tolerance 
stackup. This is facilitated by following the dimensioning and tolerancing system of ANSI-Y-14.5. 

4.2.6 Limitations 


A moderate amount of training and practice is required to effectively use standard dimensioning 
and tolerancing. 


4.3 Tolerance Stackup Analysis 


4.3.1 Description 

Tolerance stackup analysis determines if a form, fit, or function problem exists when 
manufacturing tolerances combine in a finished part or assembly. Tolerance stackup analysis is typically 
performed by either assuming worst-case allowable dimensions, or by using statistical analysis of 
tolerances. 

4.3.2 Application 

Tolerance stackup analysis is typically performed in phase C or D. This technique is used to 
determine the possibility or probability of having form, fit, or function problems with a design, or to 
determine a tolerance or dimension necessary to avoid form, fit, or function problems. 

4.3.3 Procedures 


Three typical methods for tolerance stackup analysis are: 

(1) Worst-case tolerance stackup analysis, used to determine size or position if all applicable 
dimensions occur at the worst-case extremes of the tolerance zones simultaneously. 
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(2) Statistical analysis of tolerances, used where the expected standard deviations of tolerances 
are combined to determine the probability of a final tolerance. 4 4 

(3) Design using simulation methods, where a computer is used to do a Monte Carlo analysis 
of the possible combinations of tolerances 4 5 


4.3.4 Example 

In the following hypothetical O-ring joint assembly (fig. 4-2), the tolerances of each component 
are shown in figure 4-3. Find the maximum tolerance stackup possible to obtain the minimum squeeze, 
and the probability that the squeeze will be less than 0.035. The nominal squeeze is 0.050 inches. 



Figure 4-2. O-ring joint. 



Figure 4-3. O-ring joint components. 


The probability of the squeeze being less than 0.035 is obtained by finding the distance from the 
mean (in terms of standard deviations) that this condition represents. The standard deviation is assumed 
to be one third of the tolerance on the parts (this means all parts will fall within 3 standard deviations of 
the nominal dimension) and is therefore: 

Component standard deviation = 0.010/3 = 0.0033 
O-ring standard deviation = 0.005/3 = 0.00167 
and by summation of squares, 

system standard deviation = (2(0.0033) 2 + (0.00167) 2 )- 5 = 0.005. 

For a squeeze of 0.035, the distance (in standard deviations) from the mean (z) is 

z = (0.035-0.050)/0.005 = -3.0. 
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Using a table for the normal distribution function, the area under the half curve for z = ±3 is 
0.4987. Since this is a one-sided question (no interest in the squeeze being 0.065), the area under the 
curve beyond z = 3 is (0.5-0.4987) = 0.0013. This value is interpreted as 0.13 percent probability that 
the squeeze on the O-ring will be 0.035 or less. 

A close look at the example above will show that more sources of variation are possible than 
those considered. For example, the surfaces compressing the O-ring may not be flat or normal to the 
direction of squeeze. Also, position tolerances are often determined at maximum material condition, thus 
position can vary more when not at maximum material condition. It can be extremely cumbersome to 
perform a statistical analysis of all the possible variations on some assemblies, so software exists to 
perform the statistical analysis. A typical example of software is the “Variation Simulation Analysis” 4 5 
that uses Monte Carlo methods to simulate the possible ways that the tolerances can stack up. 4 3 44 The 
results can be used to determine probabilities that certain overall tolerances will exceed a critical value, 
or which tolerances are most important to form, fit, or function. 

4.3.5 Advantages 

Worst-case tolerance analysis can simply determine the envelope of possible form, fit, and func- 
tion. Statistical analysis can show that, even if exceeding a requirement is possible, it may be extremely 
unlikely. 

4.3.6 Limitations 


Worst-case tolerance analysis is conservative, in that when many tolerances combine, it becomes 
increasingly unlikely that all dimensions will be worst-case simultaneously. Statistical tolerance analysis 
usually assumes a normal distribution of dimensions in the tolerance zone, which may be unrealistic. In 
addition, care must be exercised when combining tolerances, in that: 

(1) If some tolerances are much smaller than others, their inclusion in tolerance stackup 
analysis is superfluous. Consideration of significant digits may be helpful, e.g., a 0.030 
tolerance may have a smallest unit of measurement greater than a 0.0005 tolerance. 

(2) It may be superfluous to combine tolerances from different manufacturing processes, e.g., 
machining and casting. 


4.3.7 Bibliography 

Craig, M.: “Managing Variation by Design Using Simulation Methods.” Applied Computer Solutions, 
Inc. 
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5. GRAPHICAL DATA INTERPRETATION TOOLS 


There are numerous excellent texts on the appropriate use of graphical data interpretation tools. 
While this section lists and briefly discusses some of the available tools, the neophyte reader is advised 
to read and utilize standard handbook references when using these techniques in problem solving to 
avoid misuse and error. This toolbox is to provide knowledge of the existence of these techniques, and 
references for their appropriate application. 

One way to analyze data is by graphical interpretation. The analysis can be used to monitor 
performance, identify relationships, and reveal the most important variables in a set of data. The scatter 
diagram, section 5.1, makes it possible to determine if any relationship exists between two variables. 
The control chart, section 5.2, monitors the performance of a process with frequent outputs. Control 
charts are useful in trend analysis, section 8, and statistical process control, section 7.14. The bar chart 
compares quantities of data to help identify distribution patterns. This chart is discussed in section 5.3. 

One of the most common data displays is the time-line chart, section 5.4. This chart displays 
changes over time. Sorting data that share a common characteristic into different groups is often 
accomplished with a stratification chart. This chart is discussed in section 5.5. A Pareto chart, section 
5.6, is used typically when there is a need to know the relative importance of data or variables. This 
chart will also identify the problems, causes, or conditions that occur most frequently. A histogram, 
section 5.7, is a bar chart that shows a dispersion of data over a specified range. This type of chart is 
commonly used in presentations to make data easier to interpret. 

A summary of the advantages and limitations of each tool or methodology discussed in this 
section is presented in table 5-1. 


5.1 Scatter Diagram 


5.1.1 Description 

Scatter diagrams, also called XY graphs, plot raw data and allow the analyst to determine if any 
relationship exists between two variables. No interpretation of the data should be attempted, but 
correlations can be inferred. 5 1 

5.1.2 Application 

The graphic display of the scatter diagram can help one determine possible causes of problems, 
even when the connection between two variables is unexpected. The direction and compactness of the 
cluster of points gives a clue as to the strength of the relationship between the variables. The more that 
this cluster resembles a straight line, the stronger the correlation between the variables. The scatter 
diagram technique is best applied in phase E. 

The scatter diagram displays one variable on the horizontal (X) axis and the other variable on the 
vertical (Y) axis. If there is a correlation between the two variables, positive or negative, it can be 
assumed if the data from one are changed, then this will effect the data from the other. 5 - 2 
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Table 5-1. Graphical data interpretation tools and methodologies. 


Tool or Methodology 

Section 

Advantages 

Limitations 

Scatter diagram 

5.1 

(1) The general relationship between two variables can 
be determined at a glance. 

(2) The graph can help determine a possible cause(s) of 
problems by looking at correlations. 

(1) The choice of scale for the graph can distort the 
data, thus possibly giving the appearance of a 
correlation that is better or worse than reality. 

(2) The correlation does not prove a cause-and-effect 
relationship. 

Control chart 

5.2 

(1) The control chart helps one understand the 
capabilities of the process. 

(2) The chart can prevent tampering with processes that 
are under statistical control. 

(3) The chart monitors the effects of process changes 
that are aimed at improvement. 

(4) Control charts can be used without extensive 
knowledge of statistics. 

(1) The control chart tells only if the process is in 
control. 

(2) The underlying causes are not determined. 

Bar chart 

5.3 

(1) The bar chart tells its story at a glance. 

(2) It makes graphic comparisons of quantity easy to 
see. 

A bar chart is limited in the number of data categories 
that can be displayed at one time. 

Time-line chart 

5.4 

(1) The time-line chart shows a “moving picture” of 
fluctuations over time. 

(2) Defect rates can be plotted on time lines in order to 
identify trends. 

The time-line chart shows the direction of change but it 
gives no indication as to the reason for the change. 

Stratification chart 

5.5 

The approach not only produces a priority ordering of 
the problems but also identifies an improvement 
strategy. 

(1) The correct stratification variables for resolving a 
problem are generally not known prior to data 
collection. 

(2) All potentially important stratification variables 
cannot be determined without planning. 

Pareto chart 

5.6 

(1) The pareto chart helps to identify the few areas of 
concern that are most important. 

(2) The chart is useful in analyzing defect data. 

A poor pareto chart will result if the causes chosen to 
study are wrong. Some preplanning needs to be done 
before choosing categories. 

Histograms 

5.7 

(1) A histogram helps identify changes in a process as 
the data changes. 

(2) A histogram helps establish standards for a process. 

A histogram is not a good tool for computing process 
capability. 


5-2 



































5.1.3 Procedures 


As described in reference 5.2, a scatter diagram is prepared in the following manner: 

(1) Collect the two selected variables of each occurrence. 

(2) Draw the horizontal and vertical scales with equal length. 

(3) The dependent variable, the one that you can have an effect on, is assigned to the vertical 
(Y) axis. The independent variable is assigned to the horizontal (X) axis. Set the scale 
intervals and label. 

(4) Plot each data point. 

(5) A possible relationship can be determined by visual inspection of the graph. 

5.1.4 Example 

As adapted from reference 5.3, an aptitude test was given to 10 employees and the scores were 
then compared to the production levels of these employees over a certain time period. The scatter 
diagram, example shown in figure 5-1, would show if there were any relationship between the test scores 
and the production levels. 


Employee 

Test Score 

Production Level 

1 

27 

120 

2 

13 

80 

3 

8 

60 

4 

37 

150 

5 

32 

135 

6 

10 

70 

7 

17 

95 

8 

22 

105 

9 

6 

50 

10 

7 

55 


This plot shows that the higher test scores result in higher production levels. 

5.1.5 Advantages 

(1) The general relationship between two variables can be determined at a glance. 

(2) The graph can help determine a possible cause of problems by looking at correlations. 

5.1.6 Limitations 

(1) The choice of scale for the graph can distort the data, thus possibly giving the appearance 
of a correlation that is better or worse than reality. 

(2) The correlation does not prove a cause-and-effect relationship. 
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Figure 5- 1 . Scatter diagram example. 

5.2 Control Chart 


5.2.1 Description 

A control chart monitors the performance of a process with frequent outputs. The chart shows a 
pictorial representation of an ongoing process and determines whether or not the process is performing 
within acceptable parameters. The control chart is based on four concepts: 

(1) All processes change with time. 

(2) Individual points of the process are unpredictable. 

(3) A stable process changes randomly, and groups of points from a stable process tend to fall 
within predictable bounds. 

(4) An unstable process does not change randomly, and when changes occur they are generally 
out of the range of normal operations. 5 2 


5.2.2 Application 

The control chart technique is best performed in phase E. As described in reference 5.2, control 
charts are used to show the variation of several variables including average ( X ) and range ( R ) as well as 
the number of defects (PN), percent defective (P), defects per variable unit (U), and defects per fixed 
unit (C). 

The upper control limits (UCL) and lower control limits (LCL) should not be confused with 
specification limits. The control limits show the natural change of a process, such that points within the 
limits generally indicate normal and expected change. Points that are outside of the control limits reveal 
that something has occurred that requires special attention because the points are outside of the built-in 
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systematic cause of change in the process. One point that is outside of the control limits does not mean 
the process is out of control but it should be explained. 

The control chart can to be used continuously to determine whether the process remains within 
established control limits. As new points are added, the chart can be monitored for points that may fall 
outside of the limits and require causes to be identified. 

Control charts are used in performing statistical process control (SPC) (sec. 7.14) and trend 
analysis (sec. 8.). 

5.2.3 Procedures 

As described in reference 5.2, a control chart (fig. 5-2) is constructed in the following manner: 

(1) Determine the control limits to show the expected change of the process. 

(2) Gather data. 

(3) Plot the data on the control chart to evaluate performance and identify the points outside of 
the control limits. 

(4) Determine why points are outside of the control limits. 

(5) Find ways to identify causes of problem points, reduce the normal variation, and improve 
the mean. 

5.2.4 Example 



Figure 5-2. Control chart example. 
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5.2.5 Advantages 

(1) The control chart helps the analyst understand the capabilities of the process. 

(2) The control chart can prevent tampering with processes that are under statistical control. 

(3) The control chart monitors the effects of process changes that are aimed at improvement. 

(4) Control charts can be used without extensive knowledge of statistics. 

5.2.6 Limitations 

(1) The control chart tells only if the process is in control. 

(2) The control chart does not indicate the underlying cause unless data on outside processes 
are included in the analysis. 

5.3 Bar Chart 

5.3.1 Description 

Bar charts show a comparison of quantities of data to help identify quantity changes. The 
quantities of data are depicted by the lengths of the bars that represent cost, percentage, or frequency of 
events. The bars may be horizontal or vertical. 5 - 2 

5.3.2 Application 

Bar charts are one of the most common types of data display and this technique is typically 
performed in phase E. Differences and similarities between and among selected categories are 
emphasized by the heights of the columns. Bar charts can show double and triple bars to compare 
different time periods or different populations. 

5.3.3 Procedures 


As described in reference 5.2, a bar chart (fig. 5-3) is constructed in the following manner: 

(1) If necessary, raw data are entered on a checklist (sec. 7.8). 

(2) List the categories across the horizontal scale at the bottom. 

(3) Label the quantities on the vertical scale at the left. Make sure the scale is broad enough to 
include the highest and lowest value in each category. 

(4) Draw the bar according to the quantity of each category. 

(5) Give the bar chart a legend to identify different colors or patterns. 
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5.3.4 Example 


Sale of Household Appliances 
1980 versus 1990 
(in millions) 
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Figure 5-3. Bar chart example. 


□ 1980 

□ 1990 


5.3.5 Advantages 

(1) The bar chart tells its story at a glance. 

(2) The bar chart makes graphic comparisons of quantity easy to see. 

5.3.6 Limitations 

A bar chart is somewhat limited in the number of data categories that can be displayed at one 

time. 
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5.4 Time-Line Chart 


5.4.1 Description 

The time-line chart is among the most common types of data displays. The chart graphically 
displays changes over a period of time. 

5.4.2 Application 

The time-line chart is a special case of XY plots where the independent variable is some time 
value. The chart connects data points with line segments. The line segments connecting the points on the 
chart give a clear picture of changes over time. The vertical scale is a quantity while the horizontal scale 
is divided into time intervals such as “hours,” “days,” and “weeks.” 5 - 2 This technique is best performed 
in phase E. 

5.4.3 Procedures 

As described in reference 5.2, a time-line chart (fig. 5-4) is prepared in the following manner: 

(1) Enter the raw data on a checklist (sec. 7.8). 

(2) Establish time intervals (usually hours, days, weeks, etc.) for the horizontal axis. The 
intervals should be evenly spaced and labeled. 

(3) Establish the quantities for the vertical axis and make them evenly spaced (e.g., 10, 20, 30, 
etc.) and label the axis. 

(4) Connect, with line segments, the quantities plotted for each successive interval. 

(5) If the points are difficult to read, add horizontal and vertical grids. 

(6) Title the chart to define the time period for which the data are displayed. 

5.4.4 Example 

A study was made comparing the average number of errors that were made per operator at 
different times of the day over a certain time period (fig. 5-4). 

5.4.5 Advantages 

(1) The time-line shows a “moving picture” of fluctuations over time. 

(2) Defect rates can be plotted on time lines in order to identify trends. 
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No. of Errors 



Time Intervals 


Figure 5-4. Time-line chart example. 


5.4.6 Limitations 


The time-line chart shows the direction of change but it gives no indication as to the reason for the 
change. 


5.5 Stratification Chart 


5.5.1 Description 

The term “stratification,” derived from “stratum,” is used in data analysis. Stratification is done by 
sorting data into different groups that share a common characteristic. Some common stratification 
variables are shift, operator, and machine. 

5.5.2 Application 

The stratification chart is best applied in phase E. Comparisons of different groups, units, or other types 
of strata can often lead to suggesting an improvement strategy. For example, a process is incurring a 10- 
percent defect rate with a particular product. You can stratify by vendor, lot, operator, shift, time, 
machine, etc. and compute a percent defective for each category (stratification variable). 

The data can be depicted in graphic form for easy visual interpretation. Should the data not include a 
significant problem, select other stratification variables and collect more data. The graph may show that 
one category is producing a higher defect rate than others. This does not mean the “cause” of a problem 
has been found. What has been found is where the problem is occurring the most. The cause has yet to 
be determined. 5 4 
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5.5.3 Procedures 


As described in reference 5.4, the stratification process (fig. 5-5) is performed in the following 
manner: 

(1) Choose the stratification variables. 

(2) Gather data and record the potentially important stratification variables. 

(3) Graph the data using one of a number of different tools, such as bar chart (sec. 5.3), Pareto 
chart (sec. 5.6), and histograms (sec. 5.7). 

(4) Analyze the data on the chosen stratification variables and compare to each other. 

(5) Separate the possible problem areas into special and common cause problems. 

(6) If no conclusions are found, choose different stratification variables. 

(7) Determine the strategy to improve the problem. 


History of Discrepancy Reports for a Solid Rocket Motor 
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Figure 5-5. Stratification (histogram) chart example. 
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5.5.4 Example 

Shown in figure 5-5 is a histogram of discrepancy reports for a solid rocket motor (SRM), 
stratified by components. 

5.5.5 Advantages 

The approach not only produces a priority ordering of the problems but also identifies areas for 
improvement. 

5.5.6 Limitations 

(1) The correct stratification variables for resolving a problem are generally not known prior to 
data collection. 

(2) All potentially important stratification variables cannot be determined without planning. 


5.6 Pareto Chart 

5.6.1 Description 

When there is a need to know the relative importance of data or variables (problems, causes, or 
conditions), a Pareto chart is often used. This chart helps to highlight the few data or variables that may 
be vital. The Pareto chart also helps to identify which problems, causes, or conditions are the most 
important or most frequent so they can be addressed first. 5 - 2 

5.6.2 Application 

The Pareto chart can be used to examine the “how,” “what,” “when,” “where,” and “why” 
of a suspected problem cause. This technique is typically performed in phase E. The chart is an 
illustration of the data as of a specific time period. The data are arranged in descending order with the 
most important to the left. The Pareto chart is based on the “Pareto principle” which states that a few of 
the causes often account for most of the effects. 5 - 5 Pareto charts are used in performing problem trend 
analyses (sec. 8.2). 

5.6.3 Procedures 

As described in reference 5.2, a Pareto chart (fig. 5-6) is created in the following manner: 

(1) Identify the most likely causes of a problem (take from the cause/effect diagram (sec. 7.2)). 

(2) Gather the data on causes; if necessary, use a checklist (sec. 7.8). 

(3) Summarize the numbers of observations and calculate the percentages of each cause. 

(4) Set the right vertical scale from zero to 100 percent. 

(5) Make the left vertical scale the same height as the right scale and set it from zero to the 
number of observations. 
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100% 



Figure 5-6. Pareto chart example. 


(6) The columns are drawn using the left scale. 

(7) The first point is plotted at the upper center of the first column. 

(8) Calculate and add together the percentages of cause one and two. The second point, 
corresponding to their sum, is plotted across from the right scale directly over the second 
column. The third point is found by adding the percentage of cause three to the total of one 
and two, and plot. The total of all columns added together should be 100 percent, and the 
last point is at the 100-percent point. 

(9) The plotted points are then joined with line segments. 

The chart in figure 5-6 reveals the slope is more radical over the first two bars (power supply and 
machine calibration) and this means that the majority of the problems occur in these categories, i.e., 
areas to the left of the most radical slope are the most probable problem areas. This observation is even 
more obvious when the heights of the bars are examined. 
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5.6.5 Advantages 

(1) The Pareto chart helps to identify the few areas of concern that are most important. 

(2) The chart is useful in analyzing defect data. 

5.6.6 Limitations 

A poor Pareto chart will result if the causes chosen to study are wrong. Some preplanning needs 
to be done before choosing categories. 

5.6.7 Bibliography 

Cane, V.E.: “Defect Prevention, Use of Simple Statistical Tools.” Ford Motor Company, Livonia, MI, 
1989. 

Hines, W.W., and Montgomery, D.C.: “Probability and Statistics in Engineering and Management 
Science.” John Wiley, New York, 1986. 

Wadsworth, S. and Godfrey: “Modern Methods for Quality Control and Improvement.” John Wiley, 
New York, 1986. 


5.7 Histograms 


5.7.1 Description 

Histograms are bar charts that show a dispersion of data over a specified range. This spread of 
data makes presentations easier to interpret. 51 

5.7.2 Application 

When data are plotted on histograms, many items tend to fall toward the center of the data 
distribution. Fewer items fall on either side of the center. The bars are proportional in height to the 
frequency of the group represented. Since group intervals are equal in size, the bars are of equal width. 5 4 
The histogram is best applied in phase E. 

5.7.3 Procedures 

As described in reference 5.2, a histogram (fig. 5-7) is constructed in the following manner: 

(1) Gather the data to be plotted and count the total number of data points. 

(2) Find the range of the data by subtracting the smallest data point from the largest. 

(3) The number of data bars in the graph should be limited to between 6 and 12. The width of 
each bar is determined by dividing the range of data by the selected number of bars. 

(4) Scale the groups of data on the horizontal axis. 
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(5) Scale the frequency of occurrence or the numbers on the vertical scale. 

(6) Plot the frequency of occurrence of the numbers in ascending order. 

(7) Draw the height of each bar to show the number or frequency of the group interval using 
the scale on the vertical axis. Each bar, including all data points, is the same width. 

5.7.4 Example 

The chart in figure 5-7 displays a typical histogram. 

8 
7 
6 
5 

m 4 
*3 
2 
1 
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5.7.5 Advantages 

(1) A histogram helps identify changes in a process as the data changes. 

(2) A histogram helps establish standards for a process. 

5.7.6 Limitations 

A histogram is not a good tool for computing process capability. 



Time To Complete Tasks (Hours) 

Figure 5-7. Histogram example. 
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6. STATISTICAL TOOLS AND METHODOLOGIES 


There are numerous, excellent and highly detailed texts on the appropriate use of statistical 
techniques. While this section lists and briefly discusses some of the available tools, the novice 
statistician is cautioned to read and utilize standard, handbook references when using these techniques in 
problem solving. Use solely of this text might well result in misuse and error. This toolbox does provide 
a suitable knowledge of the existence of these tools and references for their appropriate application. 

In this section, the following typical statistical processes are discussed: “student-?” (? test) 
analysis, analysis of variance (ANOVA), correlation analysis, factorial analysis, confidence analysis, 
regression analysis, and response surface methodology. 

In many of these analyses, a comparison of sample statistics and population statistics will be 
made. Here, it is assumed that population statistics would be obtained if an infinite number of specimens 
could be measured, or if the solution to a function for the probability distribution of points were 
available. Sample statistics are made from actual measurements of a sample with a finite number of 
specimens. When only sample statistics are available (as is usually the case in engineering applications), 
there is a finite probability that they are “close” to the population statistics. 6 1 

A summary of the advantages and limitations of each tool or methodology discussed in this 
section is presented in table 6-1. 


6.1 “Student-?” Analysis 


6.1.1 Description 

As described in reference 6.1, the “student-?” compares the sample statistic “?,” which is based on 
the sample mean and standard deviation, to the ?-distribution for the same sample size and a desired 
significance (probability of error). The ?-distribution is similar to the normal distribution in that with an 
infinite sample size, the ?-distribution is equivalent to the standard normal distribution. At sample sizes 
lower than infinity, the ?-distribution becomes “lower and flatter” than the normal distribution. The 

output of the ?-distribution chart is the probability (a) that ? exceeds a certain ta on the ordinate of the ?- 
distribution chart. However, usually the probability is chosen and ta is sought; a ?-distribution table is 
usually used to find ? a . 

The ?-distribution was described in 1908 by W.S. Gosset under the pen name “student,” thus the 
name “student-?” analysis. 

6.1.2 Application 

“Student-?” analyses, as described in reference 6.2, are used when sample sizes are low for the following 
functions: 

(1) Determine if a sample mean is equivalent to a population mean within a given probability 
of error. 

(2) Determine if two sample means are equivalent to each other within a given probability of 
error. 

This technique is typically applied in phase D but may also be performed in phase C or E. 


6-1 



Table 6-1. Statistical tools and methodologies. 


Tool or Methodology 

Section 

Advantages 

Limitations 

“Student-?” analysis 

6.1 

The procedure is relatively simple to apply. 

The parent distribution must be reasonably close to a 
normal distribution. 

ANOVA 

6.2 

Sources of variation can be found, random variation 
isolated, or any chosen source of variability isolated. 

The processes are time-consuming and often 
approximate. 

Correlation analysis 

6.3 

The analysis is quite simple. 

A straight-line relationship is assumed. 

Factorial analysis 

6.4 

Sources of variation can be found, random variation 
isolated, or any chosen source of variability isolated. 
Also, interactions between variables can be isolated, and 
large numbers of variables can be solved. 

The processes in factorial analysis are more time- 
consuming than the analysis of variance. A full factorial 
analysis does not solve for exponential or polynomial 
effects. The fractional factorial analysis does not solve 
for all effects and higher order effects separately. 

Confidence/reliability 
determination and analysis 

6.5 

This analysis can give a realistic probability of whether 
or not a process may yield a value which is above or 
below a requirement. 

A sample statistic must be known or assumed, such as 
the population standard deviation, before an analysis can 
be performed. 

Regression analysis 

6.6 

A mathematical relationship can be determined, by hand 
or computer, when the relationship is not obvious by 
inspection. 

If the data are discrete (e.g., integer data), the actual line 
generated will only approximate the actual relationship. 

Response surface methodology 

6.7 

A mathematical relationship can be determined, by hand 
or computer, when the relationship is not obvious by 
inspection. 

If the data are discrete (e.g., integer data), the actual line 
generated will only approximate the actual relationship. 
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6.1.3 Procedures 


The use of a /-test for determining if a sample mean is equal to a chosen population mean will be 
shown here. 

(1) Determine the target mean and significance level desired. 

(2) Develop null and alternate hypotheses for the problem being investigated. If it is desired to 
prove that the sample mean is on one particular side of the population mean, the null 
hypothesis is that the sample and population mean are equal. The alternate hypothesis is 
that the sample mean is on the particular side of the population mean. If it is desired to 
prove that the sample mean is not on either side of the population mean, the null hypothesis 
would be the same, but the two alternate hypotheses would be that the sample mean is 
above or below the population mean. This latter situation would use a “two-tailed” 
analysis. 

(3) Determine the mean and standard deviation of the sample. 

(4) Determine the t value using equation (6.1). 

t = sample mean - target mean 

■ J75 (b.l) 

sample a /( n) 


(5) Compare t with t a for the desired significance and degrees-of-freedom (DOF) (n- 1). 

If t is greater than t a , the null hypothesis is disproved, i.e., it cannot be assumed with the chosen 
confidence that the sample mean is equivalent to the target mean. For a two-tailed analysis, if t is greater 
than t( a / 2 ) (or t is less than - t( al2)), the null hypothesis is disproved. 61 

6.1.4 Example 

Pull tests of a propellant sample yielded the following strains before failure: 29, 31, 35, 34, 
and 36 percent. The nominal strain capability is 34 percent. Determine with a 0.10 significance, if the 
propellant batch is representative of the nominal propellant. Since the mean of the propellant batch could 

be =, >, or <34 percent, a two-tailed analysis will be done. Thus, a/2 will be used (0.05 significance). 

The null hypothesis will be a strain capability equal to 34 percent. 

The sample mean is 33 and the sample standard deviation is 2.915. Substituting into equation (6.1), 
t = 0.1539. From the /-distribution table for 4 DOF, t a = 2.134. 

If H a had been rejected, it could be stated that there was only one chance in ten that the null 
hypothesis was rejected when it should not have been. This is referred to as a type I error. 

If H 0 were not rejected, it could be stated that the null hypothesis could not be rejected at the 
0.10 level of significance unless the probability of a type II error is determined. The determination of the 
probability of a type II error is complicated and many texts consider it beyond their scope. 


6.1.5 Advantages 

The procedure is relatively simple to apply. 
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6.1.6 


Limitations 


The distribution of the parent population must be reasonably close to a normal distribution. 

6.1.7 Bibliography 

Crow, E.L., Davis, F.A. and Maxfield, M.W.: “Statistics Manual.” NAVORD Report 3369, NOTS 948. 

Handbook 91, “Experimental Statistics.” U.S. Department of Commerce, National Bureau of Standards. 

Mendenhall, W.: “Introduction to Probability and Statistics.” Fourth edition, Wadsworth Publishing 
Company, Belmont, CA 94002, 1975. 


6.2 Analysis of Variance 


6.2.1 Description 

ANOVA is a technique used in design of experiments (sec. 5.5) to compare sample statistics, to 
determine if the variation of the mean and variance between two or more populations are attributable to 
sources other than random variation. 6 1 


6.2.2 Application 

The ANOVA technique is typically performed in phase D but may also be performed in 
phase C or E. 

Some of the uses for analysis of variance are: 

(1) Determining if two or more processes are producing products that are consistent with each 
other. 

(2) Determine which two or more processes are different if a difference in processes is 
detected. 

(3) Eliminate one source of variation to determine the effect of the others. 

(4) Determining the significance of each factor. 

6.2.3 Procedures 

As described in reference 6.1, to determine if two or more samples have different sample 
statistics, the following methods can be used to determine if the within-sample variation is greater than 
the sample-to-sample variation. If only one source of variation is being investigated, a one way 
classification is used. A factor F (equation (6.2)) is compared to F a , a value that is related to the total 
DOF, based on the number of samples ( k ) and the sample size (/;). 


between-sample variance 
mean ofwithin-sample variance 


(6.2) 
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The DOF of the number of samples is k- 1, and the DOF of the sample size is n- 1. The total DOF 
is k*(n- 1). If F exceeds F a , then a difference exists between the samples that is not only due to random 
variation. F a is found from an F distribution table. 

Rather than determining sample statistics for each sample, approximation formulas that use sums 
and averages of squares, can be used. 


SSjbs)/ (k- 1) 
SSE! kin- 1) ’ 


(6.3) 


where SS(bs) is the sum of squares (between-sample) and SSE is the sum of squares error. The SSE is 
determined from the sum of squares total (SST) and SS(bs) by the formula 

SSE = SST - SS(bs). 

SST and SS(bs ) can be found using the formulas 

SST = E (yij) 2 - C, SS(bs ) = E (7)) 2 /n - C, 

and 

C = 7 V/(k*n) 

where yy = each data point, T = total of all data points, and 7) = total for each sample. 

If two sources of variation are being investigated, a two-way classification is used. Data can be 
arranged in blocks representing one source of variation, and one data point from each sample 
representing the other source of variation is put into each block (see example below). If two sources are 
being investigated, the following approximation equations can be used: 


and 


MS(bs\) SS(bs 1)/ a- 1 

F( bs 1) = = 

MSE SSE/ 


F{bs 2) = 


MSjbs 2) 
MSE 


SSjbsl)/ b-l 
SSE/ K{a-\){b-\) 


(6.5) 


where 


SSE = SST - SS(bs 1 ) - SS(bsl ) ; SST = E(y ( y) 2 - C; 

MS = Mean square MSE = Mean square error 

SS(bsl) = E( T,) 2 /b - C; SS(bs2) = 'L(T J ) 2 /a - C; and 

C=T 1 /(k*n), 

where a = the number of samples of one source of variation and b = the number of samples of the other 
source of variation. 
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Other methods exist to isolate more sources of variability simultaneously. The latin square 
method eliminates three sources, and the Greco-Latin method eliminates four sources. These methods 
must use n 2 observations. 

Analysis of covariance is a similar technique used when conditions (such as environmental) 
change. The effect of this change is accounted for by using regression. This involves partitioning a total 
sum of products rather than squares. 

6.2.4 Example 

In the following hypothetical example, the effect of two parameters on the variability of strain 
capability of a solid propellant will be investigated. The use of three lots of polymer (parameter A) and 
two lots of curative (parameter B) will be investigated. Six batches of propellant are mixed and tested 
with the following average results: 


Polymer 

Curative 

Percent Strain 

1 

1 

30 

1 

2 

34 

2 

1 

32 

2 

2 

36 

3 

1 

31 

3 

2 

33 


The following table is arranged with parameter A set up in columns and parameter B set up in rows: 



Curative Lot 1 

Curative Lot 2 

Total for Polymer 

Polymer lot 1 

30 

34 

64 

Polymer lot 2 

32 

36 

68 

Polymer lot 3 

31 

33 

64 

Total for curative 

93 

103 

196 


here 

C= (196) 2 /6 = 6402.67, 

SST= 30 2 + 34 2 + 32 2 + 36 2 + 31 2 + 33 2 - 6402.67 = 6426 - 6402.67 = 23.33, 
SS(bsl) = (64 2 + 68 2 + 64 2 )/2 - 6402.67 = 6408 - 6402.67 = 5.33, 

SS(bsl) = (93 2 + 103 2 )/3 - 6402.67 = 16.67, 

MS(bsl) = 5.33/2 = 2.67, 

MS(bs2 ) = 16.67/1 = 16.67, 

MSE = 1.33/((3-l)(2-l)) = 0.67, 

F(sbl) = 2.67/0.67 = 4.0, and 
F(sb2) = 16.67/0.67 = 24.88. 
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Note that a = 3 is the number of sources of variation of parameter A (polymer), and b = 2 is the 
number of sources of variation of parameter B. Since F(sb 1) is less than F a for a 0.05 significance 
(F a = 5.14), polymer has no effect on strain capability. Since F(sb 1 ) for a 0.05 significance is greater 
than 5.99, strain capability is affected by the curative lot. 

6.2.5 Advantages 

Sources of variation can be found, random variation isolated, or any chosen source of variability 
isolated. 


6.2.6 Limitations 

The processes are time-consuming and often approximate. 

6.3 Correlation Analysis 

6.3.1 Description 

Correlation analysis measures the strength of a linear relationship between two sets of data. 6 - 3 

6.3.2 Application 

Correlation analysis can be used to determine if a relationship exists between two independent sets 
of variables. This technique is best performed in phase D but may also be performed in phase C or E. 


6.3.3 Procedures 

The procedures, as found in reference 6.3, for determining if two sets of data are linearly related 
is as follows: 

(1) Determine the mean of each set of data. 

(2) Determine the r value of the two sets of data using the following equation: 


Z(x, - x)(y. -y) 




( 6 . 6 ) 


where x and y are the means of the first and second set of data respectively. The value of r will 
be between -1 and 1. If r is close to 0, then no correlation is implied; if r is close to 1 (or -1) then 
a high degree of correlation is implied. 

(3) Determine the significance of the r value by using the following equation: 


z = 


(n-3) ln (l+r) 
2 n (l-r ) 


(6.7) 
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(4) Look up the z value in a standard normal distribution table to determine the probability of 
having a correlation. 


6.3.4 Example 

The following hypothetical sets of measurements were taken: 5.4, 6.2, 6.5, 7, and 7.5; and 2.3, 
2.1, 2, 1.8, and 1.6. The mean of the two sets are 6.52 and 1.96, respectively . The deviations, products, 
and squares of the deviations from the means are shown in the following. 


X 

y 

dx 

dy 

dx * dy 

dx 2 

dy 2 

5.4 

2.3 

-1.12 

0.34 

-0.3808 

1.25 

0.1156 

6.2 

2.1 

-0.32 

0.14 

-0.0448 

0.1024 

0.0196 

6.5 

2.0 

-0.02 

0.04 

-0.0008 

0.0004 

0.0016 

7.0 

1.8 

0.48 

-0.16 

-0.0768 

0.2304 

0.256 

7.5 

1.6 

0.98 

-0.36 

-0.3528 

0.9604 

0.1296 

summations 




-0.856 

2.548 

0.292 


Using equation (6.6), the r value is 0.992. Using this value for n = 5, z is -3.938, thus there is 
less than a 0.01 percent chance of these two data sets not being related. 

6.3.5 Advantages 

This analysis is simple to apply. 

6.3.6 Limitations 

A straight-line relationship is assumed. 


6.4 Factorial Analysis 

6.4.1 Description 

There are three types of factorial analysis described in this section — factorial analysis, full 
factorial analysis, and fractional factorial analysis. Factorial analysis is similar to ANOVA in that the 
analysis is based on sums of squares, however, factorial analysis further subdivides the treatment of 
sums of squares into components and can show interaction effects between parameters. 6 1 

6.4.2 Application 

Factorial analysis is used for applications similar to those for which ANOVA is used, except that 
factorial analysis deals with levels of variables. Factorial analysis is used with a small number of 
variables (e.g., two to four). Full factorial analysis is performed for more variables, but only at two 
levels for each variable. Fractional factorial analysis is used when so many variables are being 
investigated that experimenting with them is unfeasible. For example, if five variables are being 
investigated, 2 5 or 32 experiments would have to be performed. For six variables, the number would be 
64, and this is without replication. Thus, fractional factorial analysis is often economically necessary. 61 
This technique is typically performed in phase C but may also be performed in phase D or E . 
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6.4.3 Procedures 


As described in reference 6.1, factorial analysis is performed the same as analysis of variance 
except that an analysis of variance is performed for each variable against each other variable. 

The procedure for performing a full factorial analysis will be discussed here. With factorial 
analysis, 2 n factorial experiments will be performed, and to account for experimental variability, r 
replications will be performed. Here n will be the number of factors rather than the sample size (which is 
effectively two). With factorial analysis, certain computational shortcuts can be applied when only two 
levels of each variable are used, assuming straight line relationships. The following is the procedure for 
using the factorial analysis where n = 3. 

(1) Arrange the factors and magnitudes in a table such as the following: 


Table 6-2. Factorial analysis factors and magnitudes. 


A0,50,C0 

Ml 

M2 

M3 

total AO, BO, CO 

A1,50,C0 

Ml 

M2 

M3 

total A 1,50, CO 

A0,51,C0 

Ml 

M2 

M3 

total AO, 51, CO 

A1,51,C0 

Ml 

M2 

M3 

total Al, 51, CO 

A0,50,C1 

Ml 

M2 

M3 

total AO, 50,C1 

Al,B0,Cl 

Ml 

M2 

M3 

total A1,50,C1 

A0,Bl,Cl 

Ml 

M2 

M3 

total AO, 51, Cl 

Al,Bl,Cl 

Ml 

M2 

M3 

total Al, 51, Cl 


etc. where the first column represents the experimental conditions and Ml, M2, and M3 represent the 
resulting magnitudes after the experiment for replication 1, 2, and 3. The last column is the total of all 
replications of experiments for each experimental condition. 

(2) Obtain a table of effects totals by removing the middle columns in the above table. 

(3) Apply the method of Yates to this table as follows: 

a. Add n (3) columns in the place of the middle columns and three columns to the right 
side of the table (table 6-3). 

b. Add the first two totals in the totals column to get the first element in column 1 . Add 
the third and fourth totals in the totals column to get the second element in column 1. 
Continue in a like manner to get the third and fourth elements in column 1 . Obtain the 
fifth through eighth elements in column 1 the same way except that the totals are 
subtracted (first value subtracted from the second). Column 2 is constructed the same 
way from column 1 as column 1 was constructed from the totals column. Column 3 is 
constructed the same way from column 2. Column 3 is the effect totals as in analysis of 
variation. The notation in column n (3) and the sum of squares column is shortened; 

2: 1 means the first element of column 2. 

c. Add a row for the error sum of squares and error mean square, determined as in 
ANOVA. 
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Table 6-3. Factorial analysis example. 



Exp. 

Condition 

Totals frorr 
Above 

1 

2 

3 

Sum of 
Squares 

Mean of 
Squares 

F 

1 

AO, BO, CO 

fl 

tl + tl 

(tl + tl) + (t3 + tA) 

2:1 +2:2 

3:l/(r2«) 

SS 1/DOF 

MS1/SME 

2 

At, BO, CO 

t2 

t3 + tA 

(t5 + t6) + (tl + ^8) 

2:3 + 2:4 

3:l/(r2 n ) 

SS2/DOF 

MS2/SME 

3 

A0,B\,C0 

t3 

t5 + 16 

(f2 - fl) + (f4 - f3) 

2:5 + 2:6 

3:l/(r2«) 

SS3/DOF 

MS 3/S ME 

4 

A\,B\,C0 

tA 

fl + tS 

(f6 - f5) + (f8 - tl) 

2:7 + 2:8 

3:l/(r2«) 

SS4/DOF 

MS4/SME 

5 

A0,B0,C\ 

t5 

t2-t\ 

(f3 + tA) - (fl + t2) 

2:2 - 2:1 

3:l/(r2«) 

SS5/DOF 

MS5/SME 

6 

At, BO, Cl 

t6 

tA - 13 

(fl + tS) — (t5 + t6) 

2:4 -2:3 

3:l/(r2«) 

SS6/DOF 

MS6/SME 

7 

A0,B1,C1 

a 

t6 - 15 

(f4 - f3) - (f2 - fl) 

2:6 -2:5 

3:l/(r2«) 

SS7/DOF 

MS7/SME 

8 

A1,B1,C1 

f8 

00 

1 

(f8 - tl) - (f6 - f5) 

2:8 -2:7 

3:l/(r2 H ) 

SS8/DOF 

MS 8/S ME 


summation 





SSE 

SME 



To find: 

2:1 + 2:2 = (fl + /2) + (43 + f4) + (f5 + f6) + (tl + f8) 

2:3 + 2:4 = (42 - fl) + (f4 - f3) + (f6 - f5) + (48 - f7) 

2:2 - 2: 1 = (/5 + f6) + (f7 + f8) - (fl + t2) + (63 + f4) 

2:4 - 2:3 = (f6 - t5) + (f8 - f7) - (f2 - fl) + (f4 - f3) 

(4) The sum of squares column is generated by dividing the square of each adjacent element in 
column 3 by r * 2 n . 

(5) The mean of squares column is generated by dividing each adjacent element in the sum of 
squares column by its respective DOF. The DOF will be 1 for each effect, but will be n — 1 
for the error row. 

(6) Obtain each F by dividing each mean square by the error mean square. 

(7) Compare each F to F a for n— 1, DOF. If any F exceeds F a , that effect is significant. 

A fractional factorial analysis is performed the same way as the full factorial analysis except the 
analysis is split into fractions of (1/2 )P. Thus, if a five variable investigation (32 experiments) is split 
into 1/4, the number of experiments will be 2 n ~P (eight) experiments. 

6.4.4 Example 

The following are the results of a hypothetical experiment to determine if mix time, mix speed, and 
mix vacuum affects the burn rate of a propellant. Two levels of each parameter were tested as follows: 


Effect 

Parameter 

Low (0) 

High (1) 

A 

mix time 

2 hr 

3 hr 

B 

mix speed 

1 rps 

2 rps 

C 

vacuum 

no vacuum 

0.2 atm 
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Each effect was assigned a high and low level (e.g., 1 rps was assigned as low, 2 rps was 
assigned as high). The high and low levels are designated as 0 and 1, respectively. Each experimental 
condition was repeated three times with the following results: 


Exp. Condition Rep 1 


AO 50 CO 0.47 

A 1 50 CO 0.46 

AO B ICO 0.47 

A151C0 0.48 

AO 50 Cl 0.51 

A 1 50 Cl 0.49 

AO 51 Cl 0.52 

A151C1 0.50 


Rep 2 

Rep 3 

Total 

0.47 

0.52 

1.46 

0.46 

0.51 

1.43 

0.48 

0.52 

1.47 

0.50 

0.50 

1.48 

0.50 

0.54 

1.55 

0.52 

0.54 

1.55 

0.51 

0.55 

1.58 

0.52 

0.54 

1.56 


The table is repeated with the replication columns deleted and replaced with the application of three 
columns for the Method of Yates. Three additional columns are added, one for the sum of squares, one 
for the mean square, and one for the F value for each effect. 


Exp. 

Condition 

Total 

1 

2 

3 

Sum of 
Squares 

Mean of 
Squares 

DOF 

F 

A0 50 CO 

1.46 

2.89 

5.84 

12.08 

6.0803 

6.0803 

1 


A1 50 CO 

1.43 

2.95 

6.24 

-0.04 

0.000067 

0.000067 

1 

0.2977 

A0 51 CO 

1.47 

3.10 

-0.02 

0.10 

0.000417 

0.000417 

1 

1.8616 

A1 51 CO 

1.48 

3.14 

-0.02 

0.02 

0.000017 

0.000017 

1 

0.0745 

A0 50 Cl 

1.55 

-0.03 

0.06 

0.04 

0.00667 

0.00667 

1 

29.77 

A1 50 Cl 

1.55 

0.01 

0.04 

0 

0 

0 

1 

0 

A0 51 Cl 

1.58 

0 

0.04 

-0.02 

0.000017 

0.000017 

1 

0.0745 

A1 51 Cl 

1.56 

-0.02 

-0.02 

-0.06 

0.00015 

0.00015 

1 

0.669 

Replicates 

SSR 

SMR 



0.00723 

0.003615 

2 

16.138 

error 

SSE 

SME 



0.00157 

0.000224 

7 



The correction term (C) is as follows: 


_ (Sum of totals) 2 

(Number of effects) (Number of totals ) 


(6.8) 


The SST is as follows: 


SST = Sum of each individual replication squared - C. 


(6.9) 


The sum of squares treatment (SSTr) is as follows: 

SSTr = [(Sum of each individual total squared)/Number of effects] - C. (6. 10) 
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The sum of squares replication (SSR) is as follows: 


SSR = [(Sum of vertical replication total squared)/Number of rows] - C. (6.11) 

The sum of squares error (SSE) is as follows: 

SSE = SST - SSTr - SSR. (6.12) 

The sum of mean replicate (SMR) is as follows: 

SMR = SSR/DOF. (6.13) 

The sum of mean error (SME) is as follows: 

SME = SSE/DOF. (6.14) 


F a for a 0.05 confidence is 5.59, therefore effect C (vacuum) and replication have a significant 
effect on the burn rate. (The third batch of propellant may have been different for another reason such as 
contamination.) Note that since no values of F are greater than F a for any conditions where two or more 
effects are 1, then no interactions have a significant effect on bum rate. (For example, if the fourth line 
had an F greater than F(b then the interaction of mix time and mix speed would have a significant 
interaction). 

6.4.5 Advantages 

Sources of variation can be found, random variation isolated, or any chosen source of variability 
isolated. Also, interactions between variables can be isolated, and larger numbers of variables can be 
solved for. 

6.4.6 Limitations 


The processes in factor analysis are more time-consuming than the analysis of variance. A full 
factorial analysis does not solve for exponential or polynomial effects. The fractional factorial analysis 
does not solve for all effects and higher order effects separately. 


6.5 Confidence/Reliability Determination and Analysis 


6.5.1 Description 

Confidence analysis compares sample values, means, or standard deviations with population 
standard deviations to obtain a confidence interval, with a chosen significance. 

6.5.2 Application 

Confidence analysis is used to determine the interval of values that a data point could take, with a 
chosen probability of being within that interval. Confidence analysis can be used with individual points, 
means, standard deviations, regression lines, or reliability measurements such as mean time between 
failures. 6 1 This technique is typically performed in phase C or E. 
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6.5.3 Procedures 


As described in reference 6.1, the procedures for determining the confidence interval for the population 
mean, given a sample mean, will be discussed here. 

(1) Choose a confidence (a) level and obtain the a/2 term by dividing the confidence level by 

2 . 

(2) Determine, from past experience (or by adjusting the sample standard deviation), the 
population standard deviation. 

(3) Obtain the Z(oJ2) value by looking up the z value for a/2 in a normal distribution table. 

(4) The values for either end of the confidence interval is given by the equation: 

Int = m s ± z(a/2) * s p /n m (6.15) 


where Int is the low or high confidence interval value, m s is the sample mean, s p is the population 
standard deviation, and n is the sample size. For large n. the sample standard deviation can be used 
instead of the population standard deviation. 

The confidence interval for the population standard deviation, given the sample standard deviation, is 
determined in the same way as above, except equation (6.16) is used. 


Int = 


l± Z(a/ 2 )f s s l(2*n) 


TT1 


(6.16) 


where s s is the sample standard deviation. For linear regression, the confidence for the equation of the 
line is: 


Int = (a +bx a )±t a / 2 * s e *(1/ n + n(x 0 - m s ) 2 / S xx ) 1/2 (6.17) 

and for the y value: 

Int = (a + bx a ) ± t a /2 * s e *( 1 + 1/ n + n (x 0 - m s ) 2 / S xx ) 1/ " (6. 1 8) 


where 


2 2- ^vr^vv A^vv) 

se 2 =1/ (n-2)Z(yi-(a + bxi)) 2 - — r 

n(n-2)S xx 


where 

S X x =n * S- X f - ( T-t/ ) 2 , S yy = n * Lyj - (Zy;- ) 2 , and S xy = n * Zr ; -y,- - {'Lx i )*(ZY i ) . 
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6.5.4 Example 

Determine the confidence interval for insulation erosion at a station in the RSRM aft dome to 
determine if the associated compliance safety factor (CSF) may actually fall below the 1.0 minimum 
value, with a 95 percent confidence. The sample data for 18 flights (36 motors) is: 

Erosion mean 1 . 1 1 2 in 

Standard deviation 0.207 in (defined as known s p ) 
n 36 

a/2 is (l-0.95)/2 = 0.025, therefore the z(a/2) term is 1.96. Entering the above values into equation 
(6.15), the confidence interval is 1.112 ± 1.96 * 0.207/(36) 1/2 = from 1.042 to 1.182 for erosion. 

The safety factor is then calculated using the maximum erosion value and is: 

CSF = Min Ins t 

Erosion+3syf0. 1 

CSF = 3.36 = 1.766 . 

1.1 82+3(0. 207)+0.1 

So, in this instance the confidence interval is used to calculate a safety value that can be compared to a 
performance requirement. 

6.5.5 Advantages 

This analysis can give a realistic probability of whether or not a process may yield a value which 
is above or below a requirement. 

6.5.6 Limitations 


A sample statistic must be known or assumed, such as the population standard deviation, before 
an analysis can be performed. 


6.6 Regression Analysis 


6.6.1 Description 

Regression analysis is a form of curve fitting to find a mathematical relationship for a group of 
data. There are typically two types of regression: regression and multiple regression. Typical types of 
relationships which are assumed for regression include linear (straight line), polynomial, and 
exponential. A goodness of fit test is often performed to see how well the generated relationship fits the 
data. 6 - 3 


The method of least squares is probably the most frequently used method of regression. The 
equation for the method of least squares is obtained by setting the derivative equal to zero of the 
equation for the sum of the vertical distance from each y value to the mean y value. 
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6.6.2 Application 


Regression, as described in reference 6.1, is typically used for three purposes: 

(1) To find the mathematical relationship represented by a group of data points. 

(2) To determine if the magnitude of a measurement is increasing or decreasing with time or 
event. 

Regression analysis is best applied in phase D but may also be applied in phase E. There are 
several methods of regression. Multiple regression will be discussed in section 6.7. The least squares 
method is a commonly used method of regression, and will be discussed here (assuming a straight-line 
relationship). The R 2 indicates the percent variation in the dependent variable that can be explained by 
the independent variable. 

6.6.3 Procedures 


As described in reference 6.3, the use of the least squares method for finding the equation of a 
line of the form 


y = a + bx. 


(6.19) 


is as follows: 


(1) Determine the mean of the X[ values ( x ) and y, values ( y ). 

(2) Determine the deviation of each Xj and y/ value. 

(3) Determine the slope of the trend line by dividing the summation of the multiple of the 
deviations by the summation of the square of the x deviations (equation (6.19)). 


£(*/ ~ x )(yj - y) 

L(Xj - x) 2 


(6.20) 


(4) Determine the y intercept by subtracting the product of the slope and the mean x value from 
the mean y value (equation (6.20)). 


a = y -(b) x . 


( 6 . 21 ) 


The intercept and slope are used in equation (6.19) for a line representing the straight-line 
relationship. If the slope (b) is negative, then a decreasing trend may be indicated. 

The explanatory power can be determined by R 2 as follows: 

(1) Determine y values for each x value using the line generated above. 

(2) Determine the deviation of each generated y value from the mean y. 
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(3) Obtain the R 2 value by dividing the sum of the square of the generated y deviations by the 
sum of the square of the actual y deviations (equation (6.21)). 


n 2 Z(gen y t - j) 2 

R — — 

Uy t - y) 


(6.22) 


A good relationship is indicated by an R 2 value close to 1. 


6.6.4 Example 

As adapted from reference 6.3, assume the set of ordered pairs (1,4), (2,5), (3,6), (4,3), (5,5), 
(6,5), (7,4), (8,6), (9,4), and (10,5). The following table shows summations, squares, and products that 
go into the equations above: 



X 

J 

(dx) 2 

(dy) 2 

(dx)(dy) 

3^ 


1 

4 

20.25 

0.49 

3.15 

4.56 

0.0196 

2 

5 

12.25 

0.09 

-1.05 

4.59 

0.0121 

3 

6 

6.25 

1.69 

-3.25 

4.62 

0.0064 

4 

3 

2.25 

2.89 

2.55 

4.65 

0.0025 

5 

5 

0.25 

0.09 

-0.15 

4.68 

0.0004 

6 

5 

0.25 

0.09 

0.15 

4.71 

0.0001 

7 

4 

2.25 

0.49 

-1.05 

4.75 

0.0025 

8 

6 

6.25 

1.69 

3.25 

4.78 

0.0064 

9 

4 

12.25 

0.49 

-2.45 

4.81 

0.0121 

10 

5 

20.25 

0.09 

1.35 

4.84 

0.0196 

summation 

55 

47 

82.5 

8.1 

2.50 


0.0817 


where dx = xi - x , dy = yi- y,yg = generated points for each x, and dyg = yg- y. Using these data, the 
mean x value is 5.5, the mean y value is 4.7, the slope (b) is 0.0303, and the y intercept (a) is 4.533. The 
equation for the line is y = 0.0303(a) + 4.533. No significant relationship is indicated for this example, 
R2 = 0.0101. Figure 6-1 shows the points and the generated line for this data. 



Figure 6-1. Fine generated with least squares method. 


6-16 



6.6.5 Advantages 

A mathematical relationship can be determined, by hand or computer, when the relationship is 
not obvious by inspection. 

6 .6.6 Limitations 


If the data are discrete, e.g., integer data, the actual line generated will only approximate the 
actual relationship. 


6.7 Response Surface Methodology 


6.7.1 Description 

Response surface methodology is a method for surface fitting, much like regression is a method 
for curve fitting. The surface can be a plane, using two independent variables and straight-line 
relationships, or it can be a more complex surface, using polynomial relationships. There are two 
typically used methods for response surface analysis — multiple regression and factorial experimentation. 
Factorial experimentation is discussed in section 6.4 . 6 1 

6.7.2 Application 

Response surface analysis is typically used for the following purposes: 

(1) To find the mathematical relationship represented by a group of data points. 

(2) To optimize independent variables for maximum or minimum results. 

This methodology is best performed in phase D or E. 

6.7.3 Procedures 

As described in reference 6.3, the least squares method of multiple regression, assuming a 
straight-line relationship, will be shown here. The basic form of the equation for a plane surface is y = a 
+ b\x i + /? 2 -V '2 + byes +... + b n x n . This equation is minimized. After setting the derivative of the equation 
for the sum of the vertical distances or £ (yi - ( a + b\x\ + bixi + 6 3.13 +... + b n x n )) 2 to zero, the 
equations for two independent variables are: 

£y = nbo + bi * £xi + 7>2 * £*2, 

£Ol *y) = bo* £*1 + b\ * £ri 2 + ^2 * £Oi * xfi, 

£0 2 *y) = bo* £r 2 + b\ * £0i * X2) + b2* £*2 2 . (6.23) 

These equations are solved simultaneously for bo ,b\, and hi- 
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Often, if the numbers are equally spaced, a set of numbers is coded. For example, the numbers 
are substituted by assuming a smaller whole number for each original number. This practice makes 
solving for the coefficients much easier with very little cost in accuracy. 

6.7.4 Example 

In the following hypothetical example, as adapted from reference 6.3, propellant was aged at 
100°, 120°, and 140° for 1, 6, and 12 mo. Mean modulus of elasticity measurements are given for three 
propellant- aging temperatures and times. The columns for a, 2 . a 2 2 , x t x 2 , x^y, and A 2 y and the bottom row 
of summations are derived from the first two columns. 


-*1 

x 2 

y 

Xl 2 

x 2 ~ 

X\X 2 

xiy 

x 2 y 

100 

1 

360 

j 10,000 

1 

100 

36,000 

360 

120 

1 

352 

1 14,400 

1 

120 

42,240 

352 

140 

1 

347 

| 19,600 

1 

140 

48,580 

347 

100 

6 

358 

! 10,000 

36 

600 

35,800 

1,548 

120 

6 

350 

j 14,400 

36 

720 

42,000 

2,100 

140 

6 

345 

| 19,600 

36 

840 

48,300 

2,070 

100 

12 

347 

| 10,000 

144 

1,200 

35,700 

4,284 

120 

12 

349 

j 14,400 

144 

1,440 

41,880 

4,188 

140 

12 

343 

j 19,600 

144 

1,680 

48,020 

4,116 

1,080 

57 

3,151 

132,000 

543 

6,840 

377,520 

19,845 


The equations for finding the constants are as follows: 
From equation (6.23), 


3,151 =9b 0 + b l 1,080 + 6,57 
377,520 = 6 0 1,080 + 6, 132,000 + 6,6,840 


= b Q 57 + 

6,6,840 + 

6,543 

3,151 

1,080 

57 

37,7520 

132,000 

6,840 

19,845 

6,840 

543 

9 

1,080 

57 

1,080 

132,000 

6,840 

57 

6,840 

543 


b\ and b 2 are calculated in the same manner. Solving the simultaneous equations (6.23), the constants are 
6 0 = 383.98, b\ = -0.25, and b 2 = -0.6117. Therefore the equation for modulus of elasticity for the 
sample propellant is 


y = 383.98 - 0.25 *x, - 0.61 17 * x 2 . 
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6.7.5 Advantages 


A mathematical relationship can be determined, by hand or computer, when the relationship is not 
obvious by inspection. 

6.7.6 Limitations 

If the data are discrete (e.g., integer data), the actual line generated will only approximate the actual 
relationship. 
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7. TOTAL QUALITY MANAGEMENT TOOLS 


This section describes several TQM tools available to the system engineer analyst. TQM is 
applied to continuously improve performance at all levels of operation, in all areas of an organization, 
using all available human and capital resources. Improvement is addressed toward such areas as cost, 
quality, market share, schedule and growth. TQM is an ongoing effort that demands commitment and 
discipline. 

A tool to assess an operation against other operations is the benchmarking technique which is 
discussed in section 7.1. The cause and effect technique relates identified problems to their causes, and 
this tool is discussed in section 7.2. 

Concurrent engineering is more of an approach to quality management than a technique and it 

is an interaction of disciplines during the design but before production. This approach is 
discussed in section 7.3. 

Three tools that attempt to improve the quality program are the cost of quality, design of 
experiments (DOE), and evolutionary operation (EVOP). The cost of quality tracks a quality program 
and attempts to identify ways to improve the program. This technique is discussed in section 7.4. Design 
of experiments varies all possible combinations of factors and levels in an attempt to obtain the optimum 
settings for a desired output. This technique is discussed in section 7.5. A methodology for improving 
quality by looking at the production process is the evolutionary operation technique, and it is discussed 
in section 7.6. 

Group consensus techniques are often applied to solve problems. Three such tools are 
brainstorming, Delphi, and nominal group technique (NGT). These techniques are discussed in sections 
7.7, 7.9, and 7.10, respectively. 

A methodology for collecting data quickly and easily in a simplified manner is the checklist 
technique. This tool is discussed in section 7.8. 

Another tool that might apply to the group consensus technique is the force field analysis. This 
methodology counts the positive and negative forces, as well as their magnitudes, that effect the results 
of a proposed solution or change in process. The force field analysis is discussed in section 7.11. 

A methodology that is applied early in a design process is the quality function deployment 
(QFD) technique which is discussed in section 7.12. This technique is used to solve problems before the 
production phase begins and thus assists in the design of competitive products. By using a chart known 
as the house of quality, priorities are given to the possible solutions as they relate to the identified 
problems. Also, the product can be benchmarked against the competition in the areas of how well the 
product stacks up against the competition as far as handling the identified problems, and how well the 
product stacks up against the competition as far as meeting the appropriate engineering standards. 

The final four tools that are discussed in this section are applied to improve a process. These 
tools are quality loss function, SPC, flowchart analysis and work flow analysis (WFA). Quality loss 
function, discussed in section 7.13, is a method of determining “loss to society” when a product is not at 
the mean but is still within specification limits. SPC, discussed in section 7.14, is a process improvement 
tool that helps identify problems quickly and accurately. The flowchart analysis, discussed in section 
7.15, pictorially represents the steps of a process thus making it easier to eliminate nonvalued steps of 
the process. Finally, the WFA, discussed in section 7.16, examines the work process for possible 
improvements in performance and the quality of work life. 

A summary of the advantages and limitations of each tool or methodology discussed in this 
section is presented in table 7-1. 
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interpolated set. 

(3) Parameters must be interpolated from within the tested data set 
rather than extrapolated beyond it. 




























Table 7-1. TQM tools and methodologies — Continued 

























































7.1 Benchmarking 


7.1.1 Description 

Benchmarking, as described in reference 7.1, is a technique used to assess how an organization, 
or process, is performing against internal guidelines, competitors, or even noncompetitors that may be 
recognized as being superior. Benchmarking helps improve a process by recognizing priorities and 
goals. The technique must be continuously applied in order to be effective because practices constantly 
change (continuous improvement) affecting strategy. If the benchmarking process is performed once and 
forgotten, then the operation may become inefficient by not keeping up with the latest industry best 
practices. 

7.1.2 Application 

The benchmarking technique is typically performed in phase E but may also be performed in 
phase A or B. This technique can be applied when it is desirable to know the strengths and weaknesses 
of an organization’s own operation. These strengths and weaknesses can then be compared to internal 
guidelines to evaluate the organization’s conformance to those guidelines. 

Benchmarking can be applied to identify the strengths for products that directly compete with the 
organization’s specific product under consideration. The manufacturers of those competing products are 
probably using the same benchmarking technique to evaluate the competitors for their product. Once the 
strengths and weaknesses of competing products are known, the company can attempt to differentiate 
their capabilities in the marketplace. 

By accomplishing this analysis, an organization can also incorporate the strengths of their 
competitors that exist in certain areas. 

7.1.3 Procedures 


As adapted from reference 7.3, the basic elements of benchmarking include the following: 

(1) Decide which process(es) or product(s) to benchmark. 

(2) Determine the criteria to benchmark, i.e., benchmark internally against established 
guidelines, benchmark against competitors, or benchmark against noncompetitors that are 
considered industry leaders. 

(3) Choose the particular characteristics of the operation or product to benchmark. 

(4) Collect data on the processes or products that are being benchmarked. 

(5) Analyze the data, prepare an action plan, and implement the plan. 

(6) Assess the results of all the changes. 

(7) Repeat the benchmarking technique, as necessary, in order to stay up-to-date with the 
applicable operation. 
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7.1.4 Example 


The following illustration, adapted from reference 7.3, shows an example of comparative bench- 
marking between one company’s process and five competitors on a scale of 1 (worse) to 10 (better). 


Better 


Organization 

Process 


Worse 


10^ 

— Competitor 2 

8^ 

— Competitor 3 

7 


6^ — 

— Competitor 1 

5 


4^ 

— Competitor 5 

3 


2+4 — 

Competitor 4 


1 


Figure 7-1. Comparative benchmarking. 


This illustration reveals that this company needs to look closely at the operations of competitors 
2 and 3 and consider implementing into their process any strengths that are discovered. This company 
should also look at those competitors rated lower on the scale and identify their weaknesses and ensure 
that those weaknesses do not exist in their operation. 


7.1.5 


Advantages 

(1) Benchmarking helps meet customer requirements. 

(2) Benchmarking helps establish goals and priorities. 

(3) Benchmarking helps determine true measures of productivity. 

(4) Benchmarking helps to attain and maintain a competitive position. 

(5) Benchmarking helps identify and maintain awareness of industry’s best practices. 


7.1.6 Limitations 

(1) The benchmarking process must be continuous in order to keep up with the latest industry 
changes. 

(2) Determining industry “best practices” is often difficult and subjective. The reviewing 
company may well bias their results based on company “wants” rather than customer 
“wants.” 
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7.2 Cause and Effect Diagrams (Also Known as Fishbone Diagrams 

or Ishakawa Diagrams) 


7.2.1 Description 

The cause and effect diagram, as described in reference 7.3, graphically represents the 
relationships between a problem (effect) and its possible causes. The development process is started in a 
group session led by a trained facilitator. The problem is stated in terms acceptable to the group. 

Possible causes are listed. The group then assigns priorities to the causes and action plans are developed. 

When a cause and effect diagram is constructed, thinking is stimulated, thoughts are organized, 
and discussions are begun. These discussions bring out many possible viewpoints on the subject. Once 
all participants reach a similar level of understanding about an issue, an expansion of ideas can then be 
examined. 

Cause and effect diagrams are developed in a form, commonly referred to as “fish,” where the 
effect is found in a box to the right which is the head of the fish. The bones of the fish show the 
organized causes. The effects and causes can be expressed in words or data. 

7.2.2 Application 

As adapted from reference 7.3, cause and effect diagrams are used to examine many different 
topics which include the following: 

(1) The relationships between a known problem and the factors that might affect it. 

(2) A desired future outcome and its related factors. 

(3) Any event past, present, or future and its causal factors. 

The cause and effect diagram is useful in examining processes such as SPC, SPC problems, (sec. 
7.14) problems. The cause and effect diagram technique is best applied in phase E but may also be 
applied in phase A or B. The technique is also useful in planning activities and brainstorming. The 
diagram is basically a controlled way of gathering and using suggestions through group consensus. 

7.2.3 Procedures 


A cause and effect diagram, as adapted from reference 7.3, is developed in the following manner: 

(1) Define the effect as clearly as is possible and place it at the head of the fish. This effect 
represents the “problem” that is being investigated. As data are collected, the effect can be 
redefined, if necessary. 

(2) The group brainstorms the causes and lists them in no particular order. These causes are 
then studied and the causes that affect these causes are identified. This will continue until 
no new causes are thought of by the group. 

(3) Once all causes are identified, list all categories, then display the categories on the diagram. 

(4) The group then prioritizes the causes by multivoting. Each member of the group lists the 
causes in order of significance. Votes are counted and a final list is written. 
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(5) The highest prioritized causes are listed on the diagram as the big bones. The next highest 
prioritized causes will be listed on the diagram as the medium bones. Finally, the least 
prioritized causes will be listed on the diagram as the small bones. 

(6) As categories and causes are included on the diagram, thinking may be stimulated and new 
causes may be identified. 

(7) Teams are then formed to research and report on preventive (i.e., proactive) measures. 

7.2.4 Examples 
Example 1: 

Assume the problem is design rework (fig. 7-2). The group fills in the probable root causes 
through “brainstorming” ideas (sec. 7.7). When complete, the group prioritizes the causes using 
multivoting. This is a technique where each person lists the causes in order of significance. Votes are 
counted and a final list is written. Teams are formed to research and report on preventive measures. In 
conclusion, a team has put their thoughts in writing and arrived at a consensus. 


GRAPHICS CHANGES SCHEDULE 

Poor \ Rushed 

i Tracking 



Working 

Outside 

Discipline 


SKILL 


DESIGN 

REWORK 


INTERFACES SPECS 

Figure 7-2. Design rework cause and effect diagram. 


Example 2: 

Figure 7-3 illustrates the resulting cause and effect diagram after the brainstorming session on 
identifying problems in receiving telephone messages. The brainstorming effort for this problem is 
covered in section 7.7.4. 

7.2.5 Advantages 

(1) The cause and effect diagram enables quality analysis groups to thoroughly examine all 
possible causes or categories. 

(2) The cause and effect diagram is useful in analyzing SPC problems. SPC detects a problem 
but can pose no solution. 
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HUMAN ERROR HARDWARE ENVIRONMENT 

Message light not turned on 


Employee forgets to sign out 
Forget to undo call forward 


Call recipient does not deliver message 

\ V 

message misplaced distribution 


Wrong message taken - incomplete message 

\ \ 

rude caller distractions 

Employee fails to look at light 
Untimely delivery of message 

Criticality of message not identified (no guidelines) 
Inability to take long detailed message 


Info not available to call recipient 

Recipient doesn't know how 
to obtain info employee 
where-abouts 


Employee does not see light — 

\ \ \_ 

Poor contrast Too small 



Peak Activity 


Number of calls 


Lack of equipment to take 
detailed/technical messages 


Lack of 

interactiveautomated 
directions to caller 


No guideline for phone 

No guidelines for message takers / system setup 

/ / / 

call pickup / ca n transfer 

/ / call coverage 


Inadequate message delivery system 

/ 


Unaware 6 No feedback of messag, 

f delivered 

ofmessage 


No identified . 


No method to 
reachemployee 
notaccessible while off- 


point of contact 



No standard guidelines 
for message takers 

/ 

long detailed messages 


Message Taker responsibilities 


METHOD 


TRAINING 


Figure 7-3. Cause and effect diagram on receiving telephone messages. 
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7.2.6 Limitations 


The development of the cause and effect diagram can be time-consuming in order to arrive at a 
group consensus. 

7.2.7 Bibliography 

Kume, H.: “Statistical Methods for Quality Improvement.” The Association for Overseas Technical 
Scholarships, 1985. 


7.3 Concurrent Engineering 

7.3.1 Description 

Concurrent engineering is the interaction of technical disciplines during the design phase to 
produce a robust design prior to production. This process is more of an engineering approach to quality 
management than a technique. 7 1 The approach attempts to link and integrate, from the outset, all 
elements of the product life cycle from conception through disposal. 

Traditionally, quality, and producibility do not review an element until after the design has been 
completed. Concurrent engineering, as described in reference 7.3, focuses on both the product and the 
process simultaneously. One method of achieving this approach is by forming multifunction teams 
consisting of engineers from several departments. This way, each department will follow the complete 
process simultaneously rather than one department examining the design and then passing it on to the 
next department and so on. 7 4 

The concurrent engineering approach has been known for many years although its use is just 
receiving widespread application in the United States. 7 - 5 

7.3.2 Application 

Because the concurrent engineering approach is used to address the product and process simulta- 
neously early in the design phase, it generally will save time and money. Through this technique, the 
team will establish design goals as well as perform trade-off analyses using such tools as QFD (sec. 

7.12) and DOE (sec. 7.5). This technique is typically performed in phase C but may also be performed in 
phase B. 

7.3.3 Procedures 


The basic elements involved in applying concurrent engineering include the following, as 
adapted from reference 7.3: 

(1) Establish multifunction teams which include members from design, quality, safety, 
marketing, manufacturing, support, etc. 

(2) Select and use design parameters that will help identify and reduce variability in the 
production process. 

(3) Use such techniques as DOE, QFD, computer-aided design, robust design, group 
technology, and value analysis to extend the traditional design approach. 
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7.3.4 Example 


Figure 7-4 illustrates an example of how concurrent engineering is applied. By using 
multifunctional teams, all phases of a product’s life cycle are simultaneously examined, thus making the 
design process more efficient in terms of both cost and schedule. 



Figure 7-4. Concurrent engineering example. 
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7.3.5 Advantages 

(1) The concurrent engineering approach can be used to shorten and make more efficient the 
design-to-development life cycle by employing the interactions of functional disciplines by 
a cross-functional team. 

(2) The approach can also be applied to reduce costs in the design-to-development life cycle. 

7.3.6 Limitations 

(1) The degree of success of this technique depends upon the degree of cooperation between 
the multifunctional team members. 

(2) Significant additional time, and associated funding, is required at the front end of a 
program to perform the coordinated planning. While time and money are saved overall 
within the effort, it is often difficult to “front-load” large tasks. 

(3) If design is pursued by projectized teams, the institutional knowledge of the organization 
becomes very difficult to capture or employ in the design decisions. 


7.4 Cost of Quality 

7.4.1 Description 

As described in reference 7.3, the cost of quality technique tracks the expense and benefit of a 
quality program. This technique can identify the unwanted cost of not doing the job right the first time as 
well as the cost of improving the job. 

Cost of quality includes all of the costs associated with maintaining an acceptable quality program, 
as well as the costs incurred as a result of failure to reach the acceptable quality level. This technique 
allows the analyst to identify costs that are often hidden. Costs will not be reduced by merely tracking the 
cost of quality but the technique may point out areas where a greater return on investment could be made. 

7.4.2 Application 

The cost of quality technique is best applied in phase E. This technique is applied to understand 
the hidden costs of a product or service and to reduce or eliminate these costs. This technique can 
identify the most significant costs and thus make it possible to prioritize the activities and/or processes 
that may need improvement. 

7.4.3 Procedures 


The cost of quality technique is applied in the following manner: 
(1) Collect cost data for the following categories; 

a. Internal failure (IF) costs 

b. External failure (EF) costs 

c. Appraisal (A) costs 

d. Prevention (P) costs 
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(2) Data are trended periodically on the standard cost of quality curve shown in figure 7-5: 



Figure 7-5. Standard cost of quality curve. 


As appraisal (reactive) and prevention efforts increase, failures decrease. A significant prevention effort 
resulting in decreased failure warrants a decrease in appraisal (i.e., audits, inspections). 

Prevention is the key. Concurrent engineering (sec. 7.3) helps achieve prevention. In some companies, 
the suggestion system and/or savings shown in process improvement measures are considered 
prevention. 

Cost of quality programs requires a cross-functional, interdepartment team to agree on what constitutes a 
cost. Programs normally consist of three phases: 

(1) Initiation. 

(2) Development. 

(3) Solidified gains. 

Failures are indirectly proportional to the appraisals/preventions. As failures decrease, manpower 
(reactive) should be decreased. Prevention costs run 2 percent or less of sales as a national average. 

There are indications that, to optimize cost-benefit relationships, it should be 10 percent. As the program 
progresses, prevention costs (proactive) should increase. 

Collection of data can be on a ROM basis and need not involve finances. Be careful not to create a 
system and become so enamored with the system that the objective of savings is obscured. 

Once data are collected and analyzed, they should be compared to a base. Examples are: 


(1) Manhours per drawing. 

(2) Direct cost per hour. 

(3) Drawings per month. 
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7.4.4 Example 

An example of a cost of quality data summary for a month is shown in table 7-2. 


Table 7-2. Month’s cost of quality. 


Cost ($) 

Subject 

P 

A 

IF 

EF 

32,000 

Drawing Errors 



X 


2,000 

T raining 

X 




78,000 

Erroneous 

Information 



X 


18,000 

Warranty Claims 




X 

10,000 

Inspection/ Audits 


X 



140,000 


2,000 

10,000 

110,000 

18,000 


The percentage breakdown is: 


Prevention 
Appraisal 
Internal failure 
External failure 


2,000/140,000 

10,000/140,000 

110,000/140,000 

18,000/140,000 


1.43 percent 
7.14 percent 
78.57 percent 
12.86 percent 
100 percent 


The total failure cost is $128,000 with only $2,000 spent on prevention. This example is 98.57 
percent reactive and only 1.43 percent proactive. 


7.4.5 Advantages 

The following advantages were adapted from reference 7.6: 

(1) The cost of quality technique helps to reveal and explain the more significant costs. 

(2) Because of increased demands for time, energy, and money, it is helpful to develop a 
quality technique whereby activities and processes that need improvement can be 
prioritized. The cost of quality technique will accomplish this. 

(3) The technique helps to reveal and explain the hidden costs of a product or service. 
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7.4.6 Limitations 


(1) If not done as part of an overall plan, the cost of quality technique can be expensive, thus 
making the goals of saving/eliminating costs unachievable. 

(2) Measurement for measurement’s sake is an easy paradigm to fall into. This technique is 
subject to misuse in this regard. 


7.5 Design of Experiments 


7.5.1 Description 

The DOE technique is a control method of selecting factors, and levels of factors, in a predeter- 
mined way and varying possible combinations of these factors and levels. Quantitative results are 
analyzed to show interactions and optimum settings of factors/levels to produce a desired output. 

This technique may make the design-to-production transition more efficient by optimizing the 
product and process design, reducing costs, stabilizing production processes, and desensitizing 
production variables. 7 3 

7.5.2 Application 

The design of experiments technique is typically performed in phase C but may also be 
performed in phase D. This technique is used to achieve a robust design as an alternative to 
experimenting in the production mode after the design has been completed. As described in reference 
7.3, the following are among the applications for the DOE analysis: 

(1) Compare two machines or methodologies. 

(2) Examine the relative effects of various process variables. 

(3) Determine the optimum values for process variables. 

(4) Investigate errors in measurement systems. 

(5) Determine design tolerances. 

7.5.3 Procedures 


As described in reference 7.3, the DOE technique is implemented as follows: 

(1) Determine all of the pertinent variables whether they be product or process parameters, 
material or components from suppliers, or environmental or measuring equipment factors. 

(2) Separate the important variables which typically number no more than four. 

(3) Reduce the variation on the important variables (including the control of interaction effects) 
through redesign, close tolerance design, supplier process improvement, etc. 

(4) Increase the tolerances on the less important variables to reduce costs. 
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7.5.4 Example 


Data (yield in pounds) were recorded in table 7-3. For example, when A was at the low (A , ) 

level (10 °F), B was at the high {B-,) level (60 psi), and C was at the low (C, ) level (30 GPM), yield was 
2.1 lbs. 


Table 7-3. 2 3 factorial design data. 



A 

L t 

A 

L 2 


Bi 

b 2 

Bi 

b 2 

Cl 

(1) 

(7) 

(6) 

(4) 

8.0 

2.1 

8.4 

2.8 

c 2 

(5) 

(3) 

(2) 

(8) 


9.9 

3.2 

8.8 

3.0 


Numbers in parenthesis are standard cell designators. Normally four readings are averaged (e.g., 8.0 at 
and C l5 is an average of four data). 

The orthogonal array is shown in table 7-4 along with the result of table 7-3. This array is used as 
a “run recipe” in the actual conduct of the experiment. For example, all factors (A, B, C ) are set at their 
low level during trial 1 . 


Table 7-4. Trial, effects, and results. 


Trial 

Main Effects 

Second-Order Effects 

Third-Order 

Results 




Effects 




A 

B 

C 

AB 

AC 

BC 

ABC 


1 

- 

- 

- 

+ 

+ 

+ 

- 

8.0 

2 

+ 

- 

- 

- 

- 

+ 

+ 

8.4 

3 

- 

+ 

- 

- 

+ 

- 

+ 

2.1 

4 

+ 

+ 

- 

+ 

- 

- 

- 

2.8 

5 

- 

- 

+ 

+ 

- 

- 

+ 

9.9 

6 

+ 

- 

+ 

- 

+ 

- 

- 

°0 

OO 

7 

- 

+ 

+ 

- 

- 

+ 

- 

3.2 

8 

+ 

+ 

+ 

+ 

+ 

+ 

+ 

3.0 


An example of the average of first order or main effects is shown using A l data and cells 1, 3, 5, 

7; thus: 


8.0 + 3.2 +9.9 + 2.1 
effects - „ 


5.80. 


An example of a second order interaction (e.g., AB) is calculated by averaging data in the cells 
where A and B are at like (F) levels and unlike (U) levels. They are: 


AB l = cells 1,5, 4,8 = 


8.0 + 9.9 +2.8 + 3.0 
4 


5.93. 
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AB V = cells 7, 3, 6, 2 = 2.1 + 3.2 + 8.4 + 8.8 

4 ' 


5.63. 


An example of the third order interaction (i.e., ABC) is calculated using cell data where the 
sum of the ABC subscripts are odd ( O ), then even (E). They are: 

In cell #1, the factor levels are: A ’s level is 1, B ’ s level is 1, and C’s level is 1. Therefore, 
1+1+1 = 3, which is an odd number. The four cells having odd sums of levels are 1,2, 3, 4. 

In cell #5, the factor levels are: A’s level is 1, B ' s level is 1, and C’s level is 2. Therefore, 
1+1+2 = 4, which is an even number. The four cells having even sums of levels are 5, 6, 7, 8. 

The calculations for all factors/levels are shown in table 7-5. 


Table 7-5. Calculation of effects. 


Summation 

Cells 

Computation 

Effect 

At 

1,3, 5.7 

(8.0+3.2+9.9+2. l)/4 

5.80 

a 2 

2, 4, 6, 8 

(8. 8+2. 8+8.4+3.0)/4 

5.75 

*t 

1,2, 5,6 

(8.0+8.8+9.9+8.41/4 

8.78 

b 2 

3, 4, 7, 8 

(3.2+2.8+2.1+3.01/4 

2.78 

C 1 

1, 4, 6, 7 

(8.0+2.8+8.4+2.11/4 

5.33 

c 2 

2, 3, 5, 8 

(8.8+3.2+9.9+3.01/4 

6.23 

ab l 

1,4, 5, 8 

(8.0+9.9+2.0+3.01/4 

5.725 

ABjj 

2, 3, 6, 7 

(8.8+3.2+8.4+2.1)/4 

5.63 

ac l 

1, 2, 7, 8 

(8.0+8.8+2.1+3.01/4 

5.48 

ACjj 

3, 4, 5, 6 

(3.2+2.8+9.9+8.41/4 

6.08 

bc l 

1, 3, 6, 8 

(8.0+3.2+8.4+3.01/4 

5.65 

BC V 

2, 4, 5, 7 

(8.8+2.8+9.9+2.11/4 

5.90 

ABC 0 

1,2, 3,4 

(8.0+8.8+3.2+2.81/4 

5.70 

ABC e 

5, 6, 7, 8 

(9.9+8.4+2.1+3.01/4 

5.85 


Steps: 


(1) Find CA vg : 

This is the overall average of all data in all cells or, 

8+9.9+2.1+3.2+8.4+8.8+2.8+3 
CAvg g 5.78. 

(2) Find an estimate of o c ; 

Estimated o c = (C A vg) 1/2 /(4) 1/2 = (5.78) 1/2 /2 = 1.202. 
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(3) Ott 7 - 7 uses upper decision lines (UDL) and lower decision lines (LDL) instead of 3a control 
limits. The reason is that a decision of significant effects must be made when the plotted 
data are beyond these lines. Ott also has a table called “exact factors for one-way analysis 
of means, H a two-sided.” H 05 is found in the table. Then calculate the 95 percent UDL and 
LDL, where a = .05, as follows: 

UDL = C Avg +H 05 (Estimated o c ) = 5.78+(1.39xl.l88) = 7.43 
LDL = C Avg -H 05 (Estimated o c ) = 5.78-(1.39xl.l88) = 4.13. 

(4) The data from table 7-5, C Avg , UDL, and LDL are graphed in figure 7-6. 


10 



2.78 

2 


1 

A, Az B, B 2 Cl C, AB L AB (J Aq, BC^K ABC ABq 

Effect 

Ligure 7-6. Lactor/level effects graph. 


Conclusion: 

The main effect of B is very significant. Going from the high to the low level decreased yield 5 
lbs. Raise B from 20 to 40 psi and run another experiment. 


Advantages 


This technique makes the design-to-production transition more efficient by optimizing the product and 
process design, reducing costs, stabilizing production processes, and desensitizing production 
variables. 7 - 3 


7.5.6 Limitations 


(1) The performance of the analysis is time-consuming and, if less than full factorial arrays are 
employed, the results will not include all parametric interactions. Preknowledge of 
interaction significance is required to support appropriate DOE technique selection. 
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(2) The DOE technique is often performed without a “verification experiment,” in which the 
predicted “optimized” parameters are tested for performance (in agreement with the 
predictions). In addition, a mistake is often made by taking the “best” experiment’s 
parameters as an optimized set rather than an interpolated set. 

(3) In order to perform the analysis, parameters must be interpolated from within the tested 
data set rather than extrapolated beyond it. 


7.5.7 Bibliography 

Bhole, K.R.: “World Class Quality.” American Management Association, 1991. 


7.6 Evolutionary Operation 


7.6.1 Description 

The EVOP technique is based on the idea that the production process reveals information on how 
to improve the quality of a process. The technique has a minimal disruption to a process and creates 
variation to produce data for analysis. Optimum control factor settings are identified for desired results. 

Small, planned changes in the operating conditions are made and the results are analyzed. When 
a direction for improvement is identified, process modifications can be made. The changes can continue 
to be made until the rate of finding improvements decreases and then the changes can be applied to 
different operating variables to identify more directions for improvement. 7 8 

7.6.2 Application 

The EVOP technique is best performed in phase E but may also be performed in phase D. This 
technique is applied to reveal ways to improve a process. An experiment may use two or more control 
factors (i.e., psi and degrees F are set) that produce a response (yield) known as response surface 
methodology (RSM) (sec. 6.7). The question that may be asked is, “What are the degrees F and psi 
settings that will produce maximum yield (pounds per batch)?” 

Evolutionary operation works well with the SPC technique (sec. 7.14) in that SPC will monitor a 
process and EVOP will reveal ways to improve the process. 

7.6.3 Procedures 


The EVOP technique is applied in the following manner: 

(1) Choose two or three variables that are likely to affect quality. 

(2) Make small changes to these variables according to a predetermined plan. 

(3) Analyze the results and identify directions for improvement. 

(4) Repeat until optimal conditions are found. 

(5) The technique can then be applied to different variables. 
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7.6.4 Example 


°F 

(Factor B) 



20 30 40 50 60 

PSI 

(Factor A) 


Figure 7-7. EVOP example. 

Cycle No 1: 

Per figure 7-7 above, select a reference point “0” (center of the box). The aim is to choose the psi 
and degrees F that yield maximum output (body of the graph). Output (yield) can be volume, length, etc. 
Corner No. 2 was maximum. Cycle No. 2 uses that comer as the reference point for the second box 
(cycle). Actually, this is a simple 2 2 factorial experiment where the low and high levels of two factors, 
i.e., degrees F and PSI were selected. Data for this example are shown in table 7-6. 

Table 7-6. EVOP cycle No. 1 data. 


RUN 

1 

TIME (A) 

TEMPERATURE (B) 

POSITION 

1 

YIELD 

20 

2 

+ 

- 

3 

30 

3 

- 

+ 

4 

40 

4 

+ 

+ 

2 

50 




0 

10 


Legend: 

= Low Level 


+" = High Level 
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Main effects are calculated for A and B and second order interaction AB as follows: 


AppFFX'T (E High Levels-E Low Levels)/2 

= [(30+50)-(20+40)]/2 = 10 

^effect = (E High Levels-E Low Levels)/2 

= [(40+50)-(20+30)]/2 = 20 

^interaction = (yield when A and B have like signs — yield when A 

and B have unlike signs)/2 

= [(20+50)-(30+40)]/2 = 0. 

The change in mean (CIM) and 2 standard error (S.E.) cannot be calculated until two cycles are 
complete. The S.E. is really a variation and encompasses 95-percent confidence within the normal curve 
The 95-percent is symmetrical with a 5-percent level of significance, or a left and right tail of 7) a 
percent each. The CIM tells when a minimum or maximum occurs by comparing the results of the four 
box corners to the reference point. 

Cycle No. 2: 

Corner No. 2 produced a maximum yield (i.e., 50) and becomes the new reference point. New 
data were recorded as shown in table 7-7. 


Table 7-7. EVOP cycle No. 2 data. 


RUN 

TIME (A) 

TEMPERATURE (B) 

POSITION 

YIELD 

1 

- 

- 

5 

26 

2 

+ 

- 

7 

32 

3 

- 

+ 

8 

38 

4 

+ 

+ 

6 

48 




0 

18 


Now, compare cycles (table 7-8). 


7-21 




Table 7-8. Comparison of EVOP cycle No. 1 and cycle No. 2 data. 


CORNER 

SUBJ ECT 

YIELD AT POSITION 


0 

5 

6 

7 

8 

A 

Sum From Cycle No.l 

10 

20 

30 

40 

50 

B 

Average From Cycle No.l 

10 

20 

30 

40 

50 

C 

New Yield Data 

18 

26 

32 

38 

48 

D 

B -C 

-8 

-6 

-2 

2 

2 

E 

New Sum = B + C 

28 

46 

62 

78 

98 

F 

New Average = E/ n 

14 

23 

31 

39 

49 


The new averages are used to calculate results. The levels of factors are determined by 
examining the cycle No. 2 box of figure 7-7. For example, when A is at the high level, use comers 6 and 
7. When A is high and B is low, use comer 7, etc. 

Aeffect = [(3 1+39)-(23+39)]/2 =4 

^effect = [(31+49)-(23+39)]/2 = 9 

^interaction = [(23+3 1 )-(39+49)J/2 = -17. 


The CIM is calculated by multiplying the reference point data by 4 (now representative of four 
comers) and letting the product be a sample, i.e., n = 1. The product is subtracted from the sum of the 
four comers and divided by 5 (i.e., four corners are n = 4 + the reference point of n = 1): 

23+31+39+49 = 142 
4x14 = 56 

86/5 = 17.2 

The standard deviation and 2 S.E. when n = 2 are calculated using standard factors developed by 
Box and Hunter. 7 9 They are K = 0.3, L = 1.41, and M = 1.26. 

For the sample standard deviation: 

s = K (corner “d” range) 
s = 0.3 (-8 to +2) = 3 

for 2 S.E. For new averages/effects: 

L(s) = 1.41x3=4.23. 
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For CIM 


M(s) = 1.26x3 = 3.78. 


Results: 


Psi limits are 4 ± 4.23 = -0.23, 8.23 
Temperature limits are 9 ± 4.23 = 4.77, 13.23. 


Conclusion: 

Since the AB interaction = -17, there is a significant impact on the maximum yield. The psi can 
be negative, positive, or nil. The temperature is borderline, but it should increase yield if it is decreased. 
Select corner No. 7 and run a third cycle. 

7.6.5 Advantages 

The following advantages are adapted from reference 7.8: 

(1) The cost of running EVOP is very low so it can be run continuously. 

(2) EVOP will increase a plant’s capacity and thus profits will also increase. 

(3) EVOP is simple and relatively straightforward. 

7.6.6 Limitations 


As described in reference 7.8, EVOP is slow, so progress is slow. If quick improvements are 
needed, then this technique is inappropriate. 


7.7 Brainstorming 


7.7.1 Description 

Brainstorming, as described in reference 7.3, is a group process wherein individuals quickly 
generate ideas on a particular problem, free from criticism. The emphasis is on the quantity of ideas, not 
the quality. In the end, the goal is to arrive at a proposed solution by group consensus. All members of 
the group are equals and each is free to express ideas openly. The technique is an excellent way of 
bringing out the creative thinking from a group. 

7.7.2 Application 

Brainstorming, as described in reference 7.1, is often used in business for such things as arriving at 
compromises during union negotiations, coming up with advertising slogans, identifying root causes of a 
problem, and finding solutions to a customer service problem. 

If done properly, bashful yet creative people can be coaxed to propose good ideas. For some important 
brainstorming sessions, a facilitator is necessary. The facilitator should be knowledgeable in the 
brainstorming process and help as much as possible in the generation of ideas but should have no stake 
in the outcome of the brainstorming session. This technique is typically performed in phase A but may 
also be performed in phase C. 


7-23 



There are three phases of brainstorming, as adapted from reference 7.3: 

(1) Generation phase — group members generate a list of ideas. 

(2) Clarification phase — the group reviews the list of ideas to make sure all members 
understand each one, discussions occur. 

(3) Evaluation phase — the group eliminates duplication, irrelevancies, or issues that are off- 
limits. 

7.7.3 Procedures 

As described in reference 7.3, conduct a brainstorming session as follows: 

(1) Clearly state the purpose of the brainstorming session. 

(2) Group members can take turns expressing ideas, or a spontaneous discussion can occur. 

(3) Discuss one topic at a time. 

(4) Do not criticize ideas. 

(5) Expand on ideas from others. 

(6) Make the entire list of ideas available for all group members to review. 

(7) After discussions and eliminations, arrive at a final proposed solution by group consensus. 

7.7.4 Example 

A group was assembled to brainstorm the causes for telephone messages not being received in a 
timely manner. Each group member was given an opportunity to express ideas on the subject. A 
spontaneous discussion developed, with some group members expanding on the ideas of others. The 
following is a list of possible causes for the telephone message problem as a result of the brainstorming 
session: 

(1) Employee not at desk 

(2) Secretary not available 

(3) Volume of calls in-house 

(4) Too many incoming calls to receptionist 

(5) Employee misses message 

(6) Employee doesn’t see light or message 

(7) Incomplete message taking 

(8) Message mishandled 

(9) Nonstandard message delivery system 

(10) Employee off- site 

(11) Criticality of message not identified 
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(12) Phone grouping not identified 

(13) Whereabouts of employee unknown by call recipient 

(14) Not utilizing available resources 

(15) Caller leaves no message 

(16) Message light not turned on 

(17) Inadequate phone system 

(18) No feedback of message delivered 

(19) Lack of procedures 

(20) No identified points of contact 

(21) No answering machines 

(22) Complicated phone system 

(23) Forgetting to undo call-forwarding 

(24) People do not know how to use phone options 

(25) Secretary does not deliver messages 

(26) Secretary not in loop 

(27) Cannot find known message in loop 

(28) Wrong message taken 

(29) Untimely delivery of message 

(30) No guidelines for message taking 

(31) Not enough phones 

(32) Not enough trunk lines 

(33) Volume of calls 

(34) Congestion at receptionist’s desk 

(35) Discontinuity at receptionist’s desk 

(36) No beepers. 

Following the brainstorming session for the causes of the problem, a cause and effect diagram 
was developed as shown in section 7.2.4, example 2. Once this was completed and more discussions 
were held, a proposed solution to the problem was presented. 

7.7.5 Advantages 

The technique takes advantage of the ideas of a group to arrive at a quick consensus. 

7.7.6 Limitations 

(1) Brainstorming only proposes a solution but does not determine one. 

(2) The technique is limited by the ability of the group to achieve consensus. 
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7.8 Checklists 


7.8.1 Description 

A checklist, as described in reference 7.3, provides a list of checkoff items that enable data to be 
collected quickly and easily in a simplified manner. The data are entered on a clear, orderly form. Proper 
use of the checklist helps to minimize errors and confusion. 


7.8.2 Application 

Checklists should be laid out in advance or data may be omitted. If done right, the checklist will 
be easy to complete and will allow for quick entry of data. One common method of data entry on a 
checklist is hash marking. 

Checklists are often used to collect data on such things as numbers of defective items, defect 
locations, and defect causes. This technique is best applied in phase E but may also be applied in phase 
A or B. 

7.8.3 Procedures 

As adapted from reference 7.3, a checklist is created in the following manner: 

(1) A group should decide ahead of time what data should be collected. 

(2) Make a draft of the checklist and ask the individuals who will fill out the form for input — 
revise as necessary. 

(3) Implement the checklist. 

(4) As data are collected, review the results and, again, revise the checklist, as necessary, to 
optimize use of the form. 

7.8.4 Example 

Table 7-9 illustrates a sample of the results of postflight hardware inspections for an imaginary 
SRM. The listed defects occurred on the corresponding motor where checked. 

7.8.5 Advantages 

(1) The checklist is quick and easy to use. 

(2) Checklists help to minimize errors and confusion. 

7.8.6 Limitations 


Time must be taken to assemble a group to decide what data should be collected. 
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Table 7-9. Motor postflight checklist. 


Defect 

Description 

Motor Number 

01 

02 

03 

04 

05 

06 

07 

08 

09 

10 

Outer Igniter Joint 
Discoloration 



V 

V 



V 

V 

V 

V 

Aft Edge GEI 
Insulation Chips 

V 

V 

V 

V 

V 

V 

V 

V 

V 

V 

Water Under 
Moisture Seal 

V 

V 

V 

V 

V 

V 





Poly sulfide 
Porosity 

V 

V 

V 

V 


V 


V 

V 

V 

Wet Soot on 
Rubber 

V 

V 

V 

V 

V 

V 

V 

V 

V 

V 

Edge 

Insulation Exposure 

V 

V 

V 

V 

V 

V 




V 

Inhibitor Erosion 

V 

V 

V 

V 

V 

V 

V 

V 

V 

V 


7.9 Delphi Technique 


7.9.1 Description 

The Delphi technique, as described in reference 7.1, is an iterative process that results in a consensus by 
a group of experts. The subject is presented to the experts. Without discussing the subject among 
themselves, the experts send their comments to a facilitator. The facilitator reviews the comments and 
eliminates those not applicable to the subject. Then, the comments are redistributed to the experts for 
further review. This iteration is repeated until a consensus is reached. 

7.9.2 Application 

The Delphi technique is best performed in phase A or B but may also be performed in phase E. This 
technique is a useful tool for finding a solution when personality differences exist between members of 
involved technical areas. A group of experts can examine the problem and, through consensus, the 
effects of the differences can be minimized. Another application for this technique is to allow all parties 
to have equal input when one personality may otherwise overpower another in a discussion. 

7.9.3 Procedures 

As adapted from reference 7.1, the Delphi technique is applied in the following manner: 

(1) Define the subject upon which the experts are to comment. 

(2) Assemble a monitor group to determine task objectives, develop questionnaires, tabulate 
results, etc. 

(3) Choose the experts, making sure they have no vested interest in the outcome. 

(4) Distribute the objectives, questionnaires, etc. to the experts for their initial set of opinions. 
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(5) The monitor team consolidates the opinions and redistributes the comments to the experts, 
making sure that the comments remain anonymous. 

(6) Repeat steps 4 and 5 until a group consensus is reached. 

7.9.4 Example 

The following example was adapted from reference 7.10: 

A fault tree was generated for an SRM igniter, documenting all conceivable failure modes 
associated with the subsystems. A sample of part of the fault tree is shown in figure 7-8. The fault tree 
was then distributed to technical experts in the solid rocket industry. The expertise represented SRM 
experience in design, structures, and processing. These experts were asked to assign subjective 
estimations of failure probabilities of each mode and cause. 



Figure 7-8. Sample of a partial igniter subsystem fault tree. 


The relative probabilities were based on a rating system which utilized a tailored version of 
MIL-STD-882C 7 11 (sec. 3.12). The experts used letters to correspond to the descriptive words as 
follows: 


Level 

Descriptive Words 

Probability 

A 

Infrequent 

0.1 

B 

Remote 

0.01 

C 

Improbable 

0.001 

D 

Very improbable 

0.0001 

E 

Almost nil 

0.00001 
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Figure 7-9 shows an example of how the technical experts assigned estimations to each failure 


level. 



D C D E B B C 


Figure 7-9. Fault tree sample with estimates assigned. 

The team that generated the fault tree then took all the responses and assigned each failure level a failure 
probability based on the letters assigned by the experts. An average was derived for each failure level 
and applied to the fault tree. This labeled fault tree was distributed to the technical experts. 

This process was repeated until a consensus on the assigned failure probabilities was arrived at by all of 
the technical experts. 

7.9.5 Advantages 

(1) This technique can be useful in eliminating personality clashes. 

(2) This technique can be useful when powerful personalities are likely to dominate the 
discussion. 

(3) Inputs from experts unavailable for a single meeting are included. 

7.9.6 Limitations 

(1) Arriving at a group consensus is time-consuming. 

(2) Assembling the group participants is difficult/time-consuming. 


7-29 




7.10 Nominal Group Technique 


7.10.1 Description 

The NGT, as described in reference 7.1, is another tool used to reach a group consensus. When 
priorities or rankings must be established, this decision-making process can be used. NGT is similar to 
brainstorming (sec. 7.7) and the Delphi technique (sec. 7.9), but it is a structured approach that is 
oriented toward more specialized problems. The group should be small (i.e., only 10 to 15 people), and 
every member of the group is required to participate. This technique is often categorized as a silent 
brainstorming session with a decision analysis process. 


7.10.2 Application 

The nominal group technique is an effective tool for producing many ideas and/or solutions in a 
short time. The technique can be used for many of the same applications as brainstorming and the Delphi 
technique. The NGT is best applied in phase A or B but may also be applied in phase E. Company 
internal technical problems can be solved, personality clashes can be overcome, and NGT can be used to 
develop new ideas to satisfy a particular problem. 7 3 

7.10.3 Procedures 

The NGT, as adapted from reference 7.1, is applied in the following manner: 

(1) Generate the idea for discussion — a facilitator presents the problem and instructions to the 
team. 

(2) The team quietly generates ideas for 5 to 15 min — no discussion is allowed and no one 
leaves until everyone is finished. 

(3) The facilitator gathers the ideas round-robin and posts them in no particular order on a flip 
chart. 

(4) The ideas are then discussed by the group; no arguments, just clarifications. Duplications 
are eliminated. 

(5) Each member of the group silently sets priorities on the ideas. 

(6) The group votes to establish the priority or rank of each idea. 

(7) The votes are tabulated and an action plan is developed. 

7.10.4 Example 

The following example was adapted from reference 7.12: 

The overall objective of this task was to define an appropriate methodology for effective 
prioritization of technology efforts required to develop replacement technologies (chemicals) mandated 
by imposed and forecast legislation. 

The methodology used was a semiquantitative approach derived from QFD techniques (sec. 

7.12). This methodology aimed to weight the full environmental, cost, safety, reliability, and 
programmatic implications of replacement technology development to an appropriate identification of 
viable candidates and programmatic alternatives. 
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A list of concerns that needed to be addressed was developed as follows in table 7-10. 


Table 7-10. Replacement technology concerns. 



Chemical Concerns 


Environmental Concerns 

(1) 

Number of sources 

(1) 

Clean air monitoring 

(2) 

Limits of resources 

(2) 

Pollution prevention 

(3) 

Availability 

(3) 

Toxic emissions 

(4) 

Stability 

(4) 

Emissions control 

(5) 

Drying ability 

(5) 

Ozone depletion potential 

(6) 

Base material compatibility 

(6) 

Chemical storage availability 

(7) 

Toxicity 

(7) 

Resource/ingredient recovery and 

(8) 

Flash Point 


recycling 

(9) 

Ease of maintenance 

(8) 

Flazardous waste management 

(10) 

Flistorical data base 



(11) 

Desirable reactivity 


Cost Concerns 

(12) 

Undesirable reactivity 

(1) 

Manpower dollars 

(13) 

Lot-to-lot variability 

(2) 

Operations dollars 

(14) 

Age sensitivity 

(3) 

Facilities dollars 

(15) 

Shelf life 

(4) 

Materials dollars 

(16) 

Bondline thickness 

(5) 

Chemical dollars 



(6) 

Other hardware dollars 


Process Concerns 

(7) 

Contracts dollars 

(1) 

Contaminants removed 

(8) 

Change of specifications dollars 

(2) 

Process steps 

(9) 

Specification verification dollars 

(3) 

Parts processed at one time 

GO) 

Change of drawings dollars 

(4) 

Required surface preparation 

(11) 

Development of procedure dollars 

(5) 

Bondline thickness 

(12) 

Emissions control equipment dollars 

(6) 

Process interaction 

(13) 

Emissions control testing dollars 

(7) 

Bondline strength required 


(8) 

Operator sensitivity 


Scheduling Concerns 

(9) 

Lot-to-lot variability 


Federal, State,and Local 

(10) 

General cleaning ability 


(1) Research 

(11) 

Surface requirements 

(2) 

Trade studies 

(12) 

Possibility of stress corrosion cracking 

(3) 

Modification in planning 

(13) 

Useful life of process part 

(4) 

Specification documentation 

(14) 

Damage caused by process 

(5) 

Requirements documentation 



(6) 

Drawing/design changes 


Regulatory Concerns 

(7) 

Production time 

(1) 

OSFIA requirements 

(8) 

Testing 

(2) 

State environmental laws 

(9) 

Vendor selection and certification 

(3) 

Local environmental laws 



(4) 

Federal environmental requirements 


Present Program Schedule 

(5) 

Future federal regulations 

(1) 

Research 



(2) 

Trade studies 


Safety Concerns 

(3) 

Modification in planning 

(1) 

Worker exposure limits 


(4) Specification documentation 

(2) 

Spill response plans 

(5) 

Requirements documentation 

(3) 

Fire response plans 

(6) 

Drawing/design changes 

(4) 

Explosion response plans 

(7) 

Production time 



(8) 

Testing 



(9) 

Vendor selection and certification 
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A necessary step for developing a QFD matrix was to assign weighting factors to all of the 
concerns. A group of 10—15 people knowledgeable in the subject of chemical replacement was 
assembled to weight the concerns as follows: 

(1) Each individual of the group, without discussion, generated ideas about the importance of 
each concern. 

(2) The facilitator collected the lists of ideas and posted them in no particular order. 

(3) The ideas were discussed to clear up any misunderstandings. 

(4) The group then voted on establishing the weighting factors on each concern. 

Table 7-11 shows the list of assigned weighting factors based on a scale of 1 (least critical) to 20 
(most critical). 

7.10.5 Advantages 

NGT is very effective in producing many new ideas/solutions in a short time. 

7.10.6 Limitations 

(1) Assembling the group participants is difficult/time-consuming. 

(2) Limiting discussion often limits full understanding of others ideas, with consequent 
divergence of weighting factors as a likely result. 


7.11 Force Field Analysis 

7.11.1 Description 

The force field analysis, as described in reference 7.1, is a technique that counts both the number 
and magnitude of positive and negative forces that effect the results of a proposed solution or change in 
process. The analysis of these positive and negative forces generally occurs after performing a 
brainstorming session (sec. 7.7) or a cause and effect diagramming session (sec. 7.2). 

This technique categorizes the identified forces as either positive or negative, and assigns a value 
(weight) to each force. All positives and negatives are added and the more positive the total, the more 
likely the proposed solution is the correct one. The more negative the total, the more likely the proposed 
solution is not correct. A strategy is then developed to lessen the negative forces and enhance the 
positive forces. 


7.11.2 Application 

The force field analysis is best applied in phase D or E. This analysis is often applied in 
determining which proposed solution, among many, will meet the least resistance. The number of forces 
should not be too high (i.e., < 20) or other more sophisticated approaches should be considered. 
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Table 7-11. Concerns with assigned weighting factors. 



Chemical Concerns 



Environmental Concerns 


(1) 

Number of sources 

7 

(1) 

Clean air monitoring 

12 

(2) 

Limits of resources 

7 

(2) 

Pollution prevention 

12 

(3) 

Availability 

14 

(3) 

Toxic emissions 

15 

(4) 

Stability 

15 

(4) 

Emissions control 

12 

(5) 

Drying ability 

14 

(5) 

Ozone depletor potential 

15 

(6) 

Base material compatibility 

17 

(6) 

Chemical storage availability 

10 

(7) 

Toxicity 

13 

(7) 

Resource/ingredient recovery 

10 

(8) 

Flash point 

13 


and recycling 


(9) 

Ease of maintenance 

8 

(8) 

Hazardous waste management 

12 

(10) 

Historical data base 

9 




(ID 

Desirable reactivity 

13 


Cost Concerns 


(12) 

Undesirable reactivity 

13 

(1) 

Manpower dollars 

17 

(13) 

Lot-to-lot variability 

11 

(2) 

Facilities dollars 

15 

(15) 

Shelf life 

9 

(4) 

Materials dollars 

14 

(16) 

Bondline thickness 

7 

(5) 

Chemical dollars 

16 




(6) 

Other hardware dollars 

14 


Process Concerns 


(7) 

Contracts dollars 

12 

(1) 

Contaminants removed 

15 

(8) 

Change of specifications dollars 

13 

(2) 

Process steps 

9 

(9) 

Specification verification dollars 

13 

(3) 

Parts processed at one time 

7 

(10) 

Change of drawings dollars 

11 

(4) 

Required surface preparation 

12 

(11) 

Development of procedure dollars 

12 

(5) 

Bondline thickness 

7 

(12) 

Emissions control equipment dollars 

15 

(6) 

Process interaction 

9 

(13) 

Emissions control testing dollars 

12 

(7) 

Bondline strength required 

9 




(8) 

Operator sensitivity 

12 


Scheduling Concerns 


(9) 

Lot-to-lot variability 

11 


Federal, State, and Local 


(10) 

General cleaning ability 

13 

(1) 

Research 

9 

(11) 

Surface requirements 

14 

(2) 

Trade studies 

8 

(12) 

Possible stress coir, crack. 

14 

(3) 

Modification in planning 

9 

(13) 

Useful life of process part 

14 

(4) 

Specification documentation 

10 

(14) 

Damage caused by process 

13 

(5) 

Requirements documentation 

11 




(6) 

Drawing/design changes 

8 


Regulatory Concerns 


(7) 

Production time 

11 

(1) 

OSHA requirements 

13 

(8) 

Testing 

14 

(2) 

State environmental laws 

14 

(9) 

Vendor selection & certification 

12 

(3) 

Local environmental laws 

14 




(4) 

Federal env. requirements 

15 


Present Program Schedule 


(5) 

Future federal regulations 

14 

(1) 

Research 

10 




(2) 

Trade studies 

11 


Safety Concerns 


(3) 

Modification in planning 

10 

(1) 

Worker exposure limits 

12 

(4) 

Specification documentation 

11 

(2) 

Spill response plans 

13 

(5) 

Requirements documentation 

11 

(3) 

Fire response plans 

14 

(6) 

Drawing/design changes 

10 

(4) 

Explosion response plans 

16 

(7) 

Production time 

11 




(8) 

Testing 

12 




(9) 

Vendor selection & certification 

11 
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Application of the force field analysis requires a proposed solution and inputs to the process. 

These inputs might come from using group consensus techniques like those discussed in earlier sections. 

Also, assigning the value (weight) to each force might also require group consensus techniques. 

7.11.3 Procedures 

The force field analysis, as adapted from reference 7.1, is performed in the following manner: 

(1) Identify the proposed solution or change in process. 

(2) Determine the forces, positive and negative, that might effect the implementation of this 
proposed solution. 

(3) Separate the forces into positive and negative lists and assign a value (weight) to each 
force. Arriving at these values may be achieved by use of a group consensus technique like 
the Delphi technique (sec. 7.9). 

(4) Establish a strategy to lessen the negative forces and enhance the positive forces. 

7.11.4 Example 

Management met to discuss the possibility of approving a suggestion to allow employees to work 

flex-time. The group identified the positive and negative forces that will affect this decision as follows: 


Positive forces 


Employees welcome change 


Increased production 


Negative forces 

Employee accessible to customer 


8 


Employees present to receive messages 


Coordinate hours to improve personal life Management aware of employee's 

schedule 


9 4 

Total : 26 Total : 18 

Figure 7-10. Force field analysis example. 


The positive forces clearly outweighed the negative forces. Management developed a strategy to 
lessen the magnitudes of the negative forces listed and thus enabled the proposal of flex-time to be 
approved. 
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7.11.5 Advantages 


The force field analysis is useful in determining which proposed solution, among many, will 
meet the least resistance. 


7.11.6 Limitations 

This technique is time-consuming in arriving at a consensus on the values (weights) of the forces, 
and is highly subjective. 


7.12 Quality Function Deployment 


7.12.1 Description 

QFD, as described in reference 7.12, is a conceptual map that provides the means for cross- 
functional planning and communications. This technique is a method of turning the customer’s voice 
into engineering language. A matrix is developed known as the “house of quality” and the main 
elements of the matrix are the WHATs (customer concerns) and the HOWs (quantifiable solutions to the 
concerns). The reason for the name “house” is because the matrix is shaped like a house and elements 
are separated into rooms, as illustrated in figure 7-11. 


WHATs 



ENGINEERING 

PARAMETERS 


BENCH 

MARKS 


Figure 7-11. House of quality. 


The other rooms of the house are defined as follows: 

(1) Relationship matrix — This is the main body of the matrix, and it is the relationship between 
each WHAT and HOW. These relationships are denoted by symbols or numbers which 
correspond to weak, medium, and strong relationships. 

Example : 1 = Weak 

3 = Medium 
9 = Strong. 
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(2) Correlation matrix — This is often called the “roof’ of the house. The roof relates each of 
the HOWs to each other and is also denoted by symbols or numbers which correspond to 
strong-positive, medium-positive, strong-negative and medium- negative. 

Example: ++ = Strong -positive 
+ = Medium-positive 
— = Strong-negative 
- = Medium-negative. 

These data become important when the consideration of trade-off factors is necessary. 

(3) Benchmarks — This room is used to assess how well the product stacks up against the 
competition. 

(4) Engineering parameters — This room is used to assess how well the product stacks up to 
applicable target values. 

Inputs to the QFD matrix will require group sessions which will involve brainstorming (sec. 7.7), cause 
and effect analysis (sec. 7.2) and other techniques that might help to gather information about customer 
requirements 7 1 


7.12.2 Application 

The QFD technique is typically performed in phase C but may also be performed in phase A or B. 
This technique may be used by every function in the producing organization and in every stage of product 
development. The main focus is to implement change during design rather than during production. 

Not only does the QFD matrix allow assessment of the product against the competition and other 
benchmarks, it also enables a prioritization of the HOWs, i.e., the results of the QFD analysis can give 
overall ratings for each quantifiable solution to the stated concerns. These ratings indicate which solutions 
are most important and need to be considered first. The most important reason for the QFD analysis is to 
identify the problem areas and the quantifiable solutions to these problems early in the design phase so 
these issues will not have to be faced during production, which could lead to delays and higher costs. 


7.12.3 Procedures 

As adapted from reference 7.13, a QFD analysis is performed as follows: 

(1) Fist and prioritize the WHATs that concern the customer. These items are generally very 
vague and require further definition. This list will be placed in rows at the left side of the 
house. Each item is weighted for importance to the customer. 

(2) Fist the HOWs that address the WHATs. This list of quantifiable solutions to the WHATs 
will be placed in columns and because the WHATs are so vague, one or more HOWs can 
relate to each WHAT. 

(3) Correlate the WHATs and HOWs. This correlation is entered into the main body of the 
matrix (relationship matrix). These relationships are weighted as noted in section 7.12.1. 

(4) Fist the benchmarks and perform an assessment. The assessment can be performed on both 
the HOWs and the WHATs. Areas for improvement can easily be noted here by comparing 
how well this product stacks up against the competition. 
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(5) Correlate the HOWs to each other as noted in section 7.12.1, step 2. 

(6) Calculate the scores of the relationships. The score for each HOW as related to each 
WHAT is determined by multiplying the weighting factor for each WHAT by the 
corresponding value in the relationship matrix. The overall ratings for the values in table 7- 
12 are calculated as follows: 


Table 7-12. QFD matrix sample calculations. 

Weighting Factors \. 


HOWs 




Solution 1 

Solution 2 

Solution 3 

Concern 1 

10 

3 

9 

1 

Concern 2 

15 

1 

9 

3 

Concern 3 

12 

9 

1 

9 

Overall Rating 


153 

237 

163 


W 

H 

A 

T 

s 


Solution 1 would have an overall rating of (10x3)+(15xl)+(12x9) = 30+15+108 = 153. 
Solution 2 would have an overall rating of (10x9)+(15x9)+(12xl)= 90+135+12 = 237. 

Solution 3 would have an overall rating of (10xl)+(15x3)+(12x9)= 10+45+108 = 163. 

This example reveals that solution 2 is the most important HOW in achieving the collective 
WHATs. 


7.12.4 Example 

A planning team for an automobile company performed a task of trying to anticipate problem 
areas in a design so they can be improved upon or eliminated early. Six customer concerns (WHATs) for 
an automobile were studied: 

(1) Good performance 

(2) Quiet 

(3) Safe 

(4) Good gas mileage 

(5) Affordable 

(6) Roomy. 
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Next, all possible solutions to these concerns (HOWs) were identified and they are: 


(1) 0-60 (s) 

(2) Fuel economy (mpg) 

(3) Horsepower 

(4) Weight (klbs) 

(5) Emissions (ppm) 

(6) Noise level (dB) 

(7) Energy absorption rate (mph) 

(8) Purchase price (k$) 

(9) Maintenance cost ($) 

(10) Head room (in) 

(11) Elbow room (in) 

(12) Leg room (in). 

This automobile company was benchmarked (sec. 7.1) against three competitors as to how well 
each company stacks up to meeting each WHAT. The benchmark rating scale used was from 1 (low) to 
5 (high). 

Engineering parameters were identified for each HOW. The first parameter for each was the 
desired parameter for this company to target. The next row delineated the current company practice for 
each parameter. A final entry for these parameters, was the percent difference between the company’s 
present level and the desired target. 

The roof was included which identified the relationships between the HOWs. The rating scale 
used was as follows: 

9 = Strong positive 
3 = Medium positive 
-3 = Medium negative 
-9 = Strong negative. 

Finally, weighting factors were given to each customer concern. That is, on a scale of 1 (low) to 
10 (high), each concern was rated for importance. All of the data were coordinated and a QFD matrix 
was developed as shown in figure 7-12. 
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Conclusions: 

(1) Looking at the overall ratings showed that the two most important solutions in achieving 
the collective concerns were the horsepower rating followed by the time taken to get from 0 
to 60 mph. 

(2) The benchmarking of this company to the three main competitors revealed that, overall, 
this company rated as well or better than the competitors. The matrix showed that this 
company could stand to improve on achieving a quiet ride, getting better gas mileage, and 
making the automobiles roomier. 


7.12.5 Advantages 

(1) The QFD technique helps organizations design more competitive, higher-quality, and 
lower-cost products easier and quicker, and is aimed primarily at the development of new 
products. 

(2) This technique helps ensure quality products and processes by detecting and solving 
problems early. 

(3) Engineering changes are reduced. 

(4) The design cycle is reduced. 

(5) Startup costs are reduced. 

(6) The voice of the customer is heard. 

(7) The technique is proactive instead of reactive. 

(8) The technique prevents problems from “falling through the crack.” 

(9) The technique is economical. 

(10) The QFD technique is easy to leam. 

7.12.6 Limitations 

(1) Assembling the group participants is difficult/time-consuming. 

(2) Even though the analysis is easy to leam, it is not easy to perform. 

7.12.7 Bibliography 
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7.13 Quality Loss Function 


7.13.1 Description 

The quality loss function technique is a Taguchi method of determining the “loss to society” 
when a product is not at the true value (i.e., mean) although it still lies within specification limits. 

As described in reference 7.14, in order to develop a function to quantify the loss incurred by 
failure to achieve the desired quality, the following characteristics must be considered: 

(1) Larger is better (LIB) — the target is infinity. 

(2) Nominal is best (NIB) — a characteristic with a specific target value. 

(3) Smaller is better (SIB ) — the ultimate target is zero. 

Traditionally, manufacturers have considered a product “perfect” if it lies between the lower and 
upper specification limits as illustrated in figure 7-13. 


LOSS 




LOSS 



- PERFECT - 




i 


LSL N USL 

Figure 7-13. Traditional view to meeting specification. 


where 


LSL = Lower specification limit 
N = Nominal 

USL = Upper specification limit. 

The problem with this approach is that when “tolerance stackup” (sec. 4.3) is considered, 
difficulties arise. If two mating parts are being manufactured, they may fall at opposite ends of their 
specific tolerance and they may not assemble properly. 
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7.13.2 Application 

The quality loss function technique is typically performed in phase E but may also be performed in 
phase D. This technique is used to improve a process, thus it can be used for productivity improvement 
measurements. For each quality characteristic there is a function which defines the relationship between 
economic loss (dollars) and the deviation of the quality characteristic from the nominal value. 714 

The application of the quality loss function L(y) also reveals indications of customer 
dissatisfaction. The further the characteristic lies from the nominal value, the more problems may arise 
and thus more customer complaints. These complaints, in turn, will lead to a financial loss. 

Of course, just because a characteristic meets the target value, it does not mean that the quality of 
the product is adequate. The specification limits may be out of line. 

7.13.3 Procedures 

As described in reference 7.14, the L(y) around the target value n is given by: 

L(y) = k(y-n) 2 (7.1) 

where 

L(y ) = loss in dollars per unit product when the quality characteristic is equal to y. 

y = the value of the quality characteristic, i.e., length, width, concentration, 
surface finish, flatness, etc. 

n = target value of y. 

k = a proportionality constant. 

By applying equation (7.1) and examining figure 7-14, it can be seen that L(y ) is a minimum at 
y = n and L(y) increases as y deviates from n. 



Figure 7-14. Quality loss function for NIB. 
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where 


Ao = consumer’s loss 
and 

Ao = tolerance. 

To apply the quality loss function equation, proceed as follows: 
(1) As given in equation (7.2): 


L(y) = k(y- nj 1 

(2) To calculate a dollar loss at some value (y), first calculate k. 

k=^. 

Ao 


(3) Calculate L(y). 


7.13.4 Example 

Determine the dollars lost at some value (y) per figure 7-15. 



Figure 7-15. Quality loss function example. 


L(y) = k(y - n) 2 


, Ao 500 500 , „ 

k = — = y = = 1.25 

Ao (20) 400 

L(y ) = 1.25 (85 - 100) 2 = 1.25 (-15) 2 = 1.25 (225) = $281.25. 


(7.2) 
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7.13.5 Advantages 

(1) The quality loss function technique is an excellent tool for evaluating loss at the earliest 
stage of the product/process development. 

(2) Useful results can be obtained quickly and at low cost. 

7.13.6 Limitations 

(1) With many manufacturers following the guidelines that their product is adequate if certain 
measurements are within the specification limits, it is difficult to convince them to apply 
this technique. 

(2) It is often very difficult to calculate the quality loss function for a given process. The 
parameter y and the relationship to any A a are generally obscure. 


7.14 Statistical Process Control 

7.14.1 Description 

SPC is a method of using statistics applied to the results of a process to control the process. 
Historical data of the performance of the process (or operation of hardware) are statistically analyzed to 
predict future performance or to determine if a process is “in control.” A process is defined as “in 
control” if there are only random sources of variation present in the process and the associated data. In 
these cases, the data can correctly be investigated with the standard methods of statistical analysis. If the 
data are not “in control,” there is some special cause of variation present in the process, and this is 
reflected in the data from that process. In these cases, this section on SPC assumes that the data 
variability is still reasonably distributed around the mean, and these procedures are applicable. If these 
procedures lead to a result of special cause variation at nearly every data point, these procedures cannot 
be correctly applied. 


7.14.2 Application 

The SPC technique is best performed in phase E. This technique is used to determine if special 
causes of variation are present in a process, or if all variation is random. In other words, SPC is used to 
ensure that a product is being produced consistently, or is about to become inconsistent. Thus, SPC can 
be used to isolate problems in a process before defective hardware is delivered. This technique can be 
used for measurement type data (real numbers) or attribute data. There are two types of attribute data — 
binomial data and poisson data. Binomial data have a given number of outcomes, e.g., three of four parts 
on an assembly can be defective. Poisson data have an unlimited number of possible outcomes, e.g., a 
yard of material may have 1, 10, or 100 flaws. 


7.14.3 Procedures 


The basic steps for conducting SPC are: 

(1) Decide how to group the data. Subgroups should be chosen to show the performance of the 
part or process of interest. For example, if a machine is producing several parts at a time, 
the parts produced at one time will be a logical subgroup. 

(2) Construct a control chart and range chart (see below). 
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(3) Determine and apply control limits to the data. 

(4) Determine if any control limits are violated. If any control limits are violated, a special 
cause is indicated. In addition to the specific control limits, the analyst must examine the 
data plot for other visual indications of special causes in the data. Any particular pattern, 
for example, would indicate a special cause is present. The use of engineering judgment is 
critical to extracting the maximum amount of data from the SPC plots. 

(5) Determine the special cause. This may require Pareto analysis or engineering judgment 
using past experience. 

(6) Implement a fix for the special cause of variation. 

(7) Plot the data to ensure that the fix has been effective. 

Control charts (sec. 5.2) are made as follows: 

(1) A plot is made of the data, in temporal order of generation, on a scatter plot. 

(2) If the data are subgrouped, the mean values of the subgroups are plotted. 

(3) A range chart is made where the range is plotted for each subgroup. If the subgroup size is 
one, a moving range chart is made. The moving range for an abscissa (“x” value) is the 
absolute value of the difference of the ordinates for the abscissas and the previous abscissa. 

(4) Determine control limits as discussed below. 

(5) Apply appropriate rules to detect a lack of control (see below). 

There are typically three control limits based on the population standard deviation of the process 
(sec. 6). If negative values of data are possible, there are six control limits. They are the mean of data 
plus or minus one, two, and three standard deviations. If one datum exceeds the mean plus three standard 
deviations, a rule 1 violation exists. If two of three data points exceed the mean plus two standard 
deviations, a rule 2 violation exists. If four of five consecutive data points exceed the mean plus one 
standard deviation, a rule 3 violation exists. If eight consecutive points exceed the mean, a rule 4 
violation exists. If negative values of data are possible, these rules apply if the values are below the 
control limit. 


For real number data, the population standard deviation is determined from the average of the 
data by the equation: 


s = R m !d 2 


(7.3) 


where 5 is the population standard deviation, R m is the mean of the subgroup ranges, and dj is a factor 
for converting the mean range to the population standard deviation. The constant c /2 can be found in 
reference 7.15. If the data are not subgrouped, the average moving range is used. The moving range is 
the difference between a data point and the preceding point. 

For binomial data, the population standard deviation is given by the equation 


s = 


( Pm ( ^ Pmd n m ) 


0.5 


(7.4) 


where p m is the mean fraction defective, and n m is the number in each sample. 
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For Poisson data the population standard deviation is given by the equation 


s = (Q a5 (7.5) 

where C is the average number of nonconformities per subgroup. 

The discussion in this section has thus far been centered on a violation of a control limit 
indicating a special cause of variation being present. The special cause itself may be a shift in the entire 
data pattern defined as a mean shift or population shift. In these cases, the limits should be modified or 
recalculated to be appropriate for the subsequent data points. A mean shift is generally attributable to an 
obvious special cause such as a change in process, material, operator, cutting head, or specification. Data 
points immediately preceding and following a mean shift should not be grouped together for any other 
analyses. 


7.14.4 Example 

A hypothetical drill jig is set up to drill five holes in a component. The five holes are of the same 
size and have the same positional tolerance. Provide a control chart showing the performance of the drill 
jig with the data below, and determine the source of any deviation from nominal hole position. Table 7- 
13 below shows the deviation from nominal hole size and position made by each drill guide for each 
part. 


Table 7-13. Nominal hole size deviations and drill guide positions. 


Part # 

Tempora 

1 

Process 

Order 

Hole 1 

Hole 2 

Hole 3 

Hole 4 

Hole 5 

Range 

Mean 

2 

1 

2 

1 

2 

3 

1 

2 

1.8 

1 

2 

1 

2 

3 

4 

3 

3 

2.6 

4 

3 

3 

3 

1 

2 

2 

2 

2.2 

5 

4 

2 

2 

2 

3 

1 

3 

2.0 

3 

5 

4 

2 

3 

2 

2 

2 

2.6 

6 

6 

2 

1 

2 

3 

1 

2 

1.8 

7 

7 

6 

3 

1 

2 

3 

5 

3.0 

10 

8 

7 

2 

2 

1 

3 

6 

3.0 

8 

9 

9 

3 

2 

2 

2 

7 

3.6 

9 

10 

10 

2 

1 

3 

4 

9 

4.0 


range 

9 

2 

2 

2 

2 

5 



mean 

4.6 

2.1 

1.9 

2.5 

2.2 

4.1 

2.66 


The mean and range for each part and each hole is shown in the table 7-13. Each part will be 
considered to be a subgroup. If the variation between holes is of primary interest, it could be better to 
treat each hole as a subgroup. However, the performance of the entire jig is of primary interest in this 
example, so each part will be treated as a subgroup. The first control chart (fig. 7-16) shows the 
performance of the jig with the mean plotted against the time -phased process order. The UCL is shown. 
The UCL is calculated using equation (7.6) to obtain the population standard deviation, multiplying it by 
3 and adding it to the mean of the mean deviation. Notice that the mean measurement is increasing for 
the last few parts, but no control limits are exceeded. 


7-46 



6 


5 

Mean 

Deviation 4 
3 
2 
1 


UCL 
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Figure 7-16. Control chart showing mean deviation for each part. 


The second chart (fig. 7-17) is a range chart that shows the mean range for each part plotted 
against part number (note that it remains in temporal order). Part number 9 exceeded the UCL range 
(UCLR). UCLR is given by the equation: 

UCLR = R m [ 1 + 3 (c/ 3 /c/ 2 )] (7.6) 

where R m is the mean range and c / 3 is a factor for converting the mean range to the standard deviation of 
the range. The constant c /3 can be found in reference 7.15. This shows that the within-group variation is 
increasing more that the group-to-group variation. 



UCLR 
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The third chart (fig. 7-18) shows a Pareto chart (sec. 5.6) where the mean deviation is plotted 
against hole number. By examination, it can be seen that drill guide position 1 is producing holes with a 
mean measurement that is higher than the other drill guide positions. 


5 
4 

Mean 

Deviation 

2 
1 

1 2 3 4 5 

Hole Number 

Figure 7-18. Pareto chart showing mean deviation for each hole guide. 



The fourth chart, figure 7-19, shows the deviation produced by hole guide 1 plotted against part 
number. By examination, it can be seen that the deviation is increasing starting with part 7. 

10 
9 
8 

Deviation T. 

6 

5 

4 

3 

2 

1 

2 14 536 7 10 89 

Part Number 

Figure 7-19. Control chart showing mean deviation for hole guide 1. 



7.14.5 Advantages 


(1) SPC is an excellent technique for determining the cause of variation based on a statistical 
analysis of the problem. 

(2) The technique improves process performance. 

(3) SPC helps identify problems quickly and accurately. 
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7.14.6 Limitations 

SPC detects problems but poses no solutions. 

7.14.7 Bibliography 
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7.15 Flowchart Analysis 


7.15.1 Description 

A flowchart, as described in reference 7.3, is a pictorial representation of the steps in a process 
where each step is represented by a block. The review of a flowchart allows the elimination of nonvalue 
added steps. When prepared by a group, the chart represents a consensus. The flowchart analysis is a 
useful tool for determining how a process works. By studying how process steps relate to each other, 
potential sources of problems can often be identified. 

Many different types of flowcharts are useful in the continuous improvement process. Flowcharts 
often used are the top-down flowchart, the detailed flowchart, and the work flow diagram. The top-down 
flowchart, figure 7-20, presents only the major and most fundamental steps in a process. This chart 
makes it easy to visualize the process in a single, simple flow diagram. Key actions associated with each 
major activity are listed below their respective flow diagram steps. A top-down flowchart can be 
constructed fairly quickly and easily. This type of flowchart is generally developed before attempting to 
produce the detailed flowcharts for a process. By limiting the top-down flowchart to key actions, the 
probability of becoming bogged down in the detail is reduced. 



• Become 
Familiar 
with TQM 

• Familiarize 
Subordinates 
with TQM 

• Develop 
Implementation 
Plan 


• Take 
First 
Step 

• Be 

Committed 

•Be 

Consistent 


• Examine 
Your Use 

• Develop 
User's Guide 


• Sell Idea 
of 

Implementation 


• Provide 
Training 
Classes 

• On the job 
Training 

• Use 
Available 
Resources 


• Listen 

to 

Customer 

• Understand 
Customer 
Needs 

• Establish 
Routine 

Communication 

with 

Customer 


Figure 7-20. Example of top-down flowchart. 
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The detailed flowchart, figure 7-21, gives specific information about process flow. At the most 
detailed level, every decision point, feedback loop, and process step is represented. Detailed flowcharts 
should only be used when the level of detail provided by the top-down or other simple flowcharts is 
insufficient to support the analysis of the process. 



Figure 7-21. Example of detailed flowchart. 


The work flow diagram (section 7.15.4) is a graphic representation of how work flows through a facility. 
This diagram is useful for analyzing flow processes, illustrating flow efficiency, and planning process- 
flow improvement. Figure 7-22 illustrates the most common flowchart symbols. 
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Activity Symbol Action that is taking place. 
Decision Symbol Yes/No Decision. 


CD 




Terminal Symbol Beginning or end of 

process. 

Flow Line - Shows direction of process flow. 
Document Symbol Indicates a document 

source. 


Data Base Symbol - Indicates a database 

source. 


On Page Connector - Indicates point elsewhere 

on a large page where 
process continues. 

Off Page Connector Indicates point on 

another page where 
process continues. 

Brick Wall - Shows obstacle beyond your control. 


Inspiration - Indicates a possible solution. 


Black Hole - Indicates a problem that 
consumes all resources. 

Dead End - Shows particular path of a process 
has no acceptable solution. 


Magic Happens Here -Indicates that, with a 

breakthrough, we can 
continue the process. 


Figure 7-22. Common flowchart symbols. 
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7.15.2 Application 


A flowchart is best applied in phase B but may also be applied in phase E. This chart is used to 
provide a picture of the process prior to writing a procedure. Flowcharts should be created, then 
procedures written to follow the flowchart. The chart should be included as an appendix in the 
procedure. Flowcharts can be applied to anything from material flow to the steps it takes to service or 
sell a product. 

7.15.3 Procedures 

A flowchart, as described in reference 7.1, is prepared in the following manner: 

(1) A development team creates a diagram that defines the scope of the task to be undertaken. 
Also identified are the major inputs and outputs. 

(2) Create a data flow diagram. Start with executive level data that are involved in the process, 
followed by department data and finally branch data. 

(3) Using the data, create an initial model. The team should walk through the process and look 
for any details that need to be clarified, added, or deleted. 

(4) Make a data dictionary. This ensures that everyone involved in the project has a consistent 
understanding of the terms and steps used. 

(5) Add the process symbols. 

(6) Revise, as necessary. 

7.15.4 Example 

The following example, figure 7-23, illustrates a work flow diagram for encountering problems 
with a copy machine. 

7.15.5 Advantages 

The following advantages are adapted from reference 7.16: 

(1) Flowcharts allow examination and understanding of relationships in a process. 

(2) Flowcharts provide a step-by-step picture that creates a common understanding about how 
the elements of a process fit together. 

(3) Comparing a flowchart to actual process activities highlights areas where policies are 
unclear or are being violated. 

7.15.6 Fimitations 


The flowchart development process can be time-consuming. 


7-52 




Figure 7-23. Work flow diagram example. 


7.16 Work Flow Analysis 


7.16.1 Description 


A WFA, as described in reference 7.1, examines the work process for possible improvements in 
performance and quality of work life. This technique is really a special case of flowcharting (sec. 7.15). 
The goal is to overcome the excuses for not changing work habits on the part of the employee as well as 
management. Such excuses are, “It has always been done this way,” and “It’s not my responsibility.” 
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7.16.2 Application 

A WFA is best applied in phase E. The analysis is performed in an employee/management 
partnership, where the goal for each party is to improve productivity as well as the quality of work life. 
The technique will work if executed by a partnership of management and employees. 

7.16.3 Procedures 

As adapted from reference 7.1, a WFA is performed in the following manner: 

(1) Collect data concerning the operation being analyzed. This can be done by observing the 
operation or asking questions, but not by reading an operations plan that would tell how the 
operation is supposed to be done. 

(2) Flowchart the process (sec. 7.15). 

(3) Research and collect ideas on how to improve the operation from any sources available. 

(4) Define the desired performance versus the actual performance. 

(5) Identify the gaps in performance and propose changes to eliminate these gaps. 

(6) Analyze these changes by using a multifunctional team. 

(7) Once the changes are agreed upon, prototype them on a small basis in a certain area or shift. 

(8) Once the bugs are ironed out and the changes are operating smoothly, implement them on a 
large-scale basis. 

(9) Flowchart the new operation and revise the operating procedure documentation to reflect 
the changes. 


7.16.4 Example 


An analysis team was assembled to analyze the food preparation process at a local fast food 
restaurant in an attempt to find areas where the operation could be run more efficiently. The steps of the 
analysis are as follows: 

(1) The first step involved observing the operation and then flowcharting the process as shown 
in figure 7-24 below. 

(2) Members of the team then observed other restaurants to find ways of improving the process. 

(3) Once the research was completed, the desired performance was identified and compared to 
the actual process. 

(4) The team, which involved management, employees, and outside consultants, then 
developed a new plan for the process. 

(5) This new process was first tried out during slow business hours to ensure the new process 
ran smoothly. 

(6) Once everyone agreed that the new process was more efficient, then it was implemented. 
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Figure 7-24. WFA example. 


7.16.5 Advantages 

The technique may increase productivity and improve working conditions. 

7.16.6 Limitations 

(1) The technique requires cooperation between employees and management to be most 
successful. 

(2) The observed operation may not be fully representative of a “typical” process that would 
occur without scrutiny. 
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8. TREND ANALYSIS TOOLS 


Trend analysis, as described in reference 8.1, is a quantitative tool used to identify potentially 
hazardous conditions and cost savings based on past empirical data. Trend analysis evaluates variations 
of data to find trends, with the ultimate objective of assessing current status and forecasting future 
events. Trend analysis can be reactive or proactive. Data examined from past events can uncover a cause 
of a problem or inefficiency in a product or operation. Also, real-time data can be tracked to detect 
adverse trends that could indicate an incipient failure or can be used to reduce discrepancies in a product 
or operation. 

Program level trending exchanges data between organizations and correlates trends from the 
various organizations to find relationships and allows integration of the trend analysis effort with any 
planned TQM effort (sec. 7), such as SPC (sec. 7.14). It also allows upper level management to forecast 
problems such as shortages, schedule delays, or failures. Finally, in starting a program level trending 
effort early in the program, data collection will be more efficient and cost-effective. 

The use of trend analysis has several benefits. Among them are: 

(1) Predicting system or process failure or violation of a process limit criterion. 

(2) Indicating that a unit can remain in service longer than anticipated or projecting the service 
life of a unit. 

(3) Eliminating the need for some hardware inspections. 

(4) Increase cost-effectiveness by reducing variability in a process. 

There are different levels of trend analysis parameter criticality based on the degree of the benefit 
derived from the results of the trend analysis for that parameter. Some parameters have a direct effect on 
system safety while others will have an impact on cost or timeliness of a process. Criticality levels have 
an impact on the amount of trending to be performed, the level to which it is to be reported, the data that 
are to be stored, and the time over which the trending is to be performed. Examples of criteria for levels 
of requirements are: 


(1) Parameters impacting personnel safety. 

(2) Parameters impacting successful system performance. 

(3) Parameters which could cause failure of a component that would not result in system 
failure. 

(4) Parameters impacting schedule of the system. 

(5) Parameters impacting delivery schedule of components. 

(6) Parameters impacting cost of manufacturing. 

Trending can be used at levels from program management to component and system production 
and vendors. Upper level management would conduct trending on program level issues, and individual 
organizations would conduct trending on issues pertinent to that organization at a component/material, 
subsystem, or system level. 
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Examples of trending activities are: 

(1) Component-receiving organizations can conduct trending on such things as would indicate 
the quality of incoming components, materials, and problems of receiving them in proper 
condition. 

(2) Manufacturing can conduct trending on component and system requirements, and 
production problems. 

(3) Test, launch, and refurbishment organizations can conduct trending on performance, time 
to conduct operations, and problems encountered. 

Some trending results will be reported to upper level management, engineering, and the 
customer, while other results would be for local use by the individual organizations. 

Five trending analysis techniques will be discussed in this section. Performance trend analysis, 
discussed in section 8.1, detects a degrading parameter prior to a potential failure as well as predicting 
future parameter values. 

Problem trend analysis, discussed in section 8.2, provides an early indicator of significant issues 
in other types of trend analysis. Other applications of this analysis are to “examine the frequency of 
problem occurrence, monitor the progress of problem resolution, uncover recurring problems, and assess 
the effectiveness of recurrence control.” 8 - 2 

A technique that provides visibility to determine the current/projected health of the human 
support element is programmatic trend analysis. This analysis is discussed in section 8.3. A technique 
that monitors the current health of support systems and forecasts support problems to enable resolution 
with minimum adverse effect is supportability trend analysis. This analysis is discussed in section 8.4. 

Finally, reliability trend analysis is discussed in section 8.5. This technique is similar to 
performance trend analysis and problem trend analysis. Reliability trend analysis measures reliability 
degradation or improvement and enables the prediction of a failure so action can be taken to avert the 
failure. 


There can be a high level of overlap for some of these types of trend analysis, depending on 
individual definitions of performance, reliability, and problems. Since many tools are useful for all types 
of trending and the trend analysis customer typically looks for known parameters, this overlap is not a 
problem. Performance, problem, and reliability trend analyses are more directly applicable to the needs 
of a system engineer, than programmatic or supportability trend analyses. However, the former two 
types of trend analysis are presented here, since results from these analyses may impact the system for 
which the system engineer is responsible. 

A summary of the advantages and limitations of each tool or methodology discussed in this 
section is presented in table 8-1. 
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Table 8-1. Trend analysis tools and methodologies. 


Tool or Methodology 

Section 

Advantages 

Limitations 

Performance trend analysis 

8.1 

(1) Detects a degrading parameter prior to a potential 
failure. 

(2) Predicts future parameter values or estimates the 
long-term range of values of influential variables. 

(3) The service life of systems or system elements can 
be predicted. 

(1) Parameter sensors may need to be installed to 
obtain trending data; this can be costly. 

(2) The operating state, output, or load, about/ 
through which a system/subsystem/component 
fluctuates, often cannot be controlled to achieve 
consistent trend data. (Data must be statistically 
stable.) 

(3) The slope and stability of the data approaching/ 
departing the recorded data point are not known 
without using a data buffer. 

(4) Data are not always easily quantifiable, limiting the 
usefulness of the technique. 

Problem trend analysis 

8.2 

(1) Provides an early indicator of significant issues in 
other types of trend analysis. 

(2) Examines the frequency of problem occurrence, 
monitors the progress of problem resolution, 
uncovers recurring problems and assesses the 
effectiveness of recurrence control. 

Candidate items should be chosen carefully because the 
analysis can be costly if performed for all potential 
problem areas. 

Programmatic trend analysis 

8.3 

This technique monitors programmatic posture and 
provides visibility to determine current/projected health 
of the human support element. 

The data collection process can be extensive because of a 
potentially large and varied number of sources. 

Supportability trend analysis 

8.4 

This technique monitors the current health of support 
systems and forecasts support problems to enable 
resolution with minimum adverse effect. 

Determining the extent of analysis and identifying the 
appropriate parameter variations that must be measured 
can be difficult. 

Reliability trend analysis 

8.5 

This technique measures reliability degradation or 
improvement and enables the prediction of failures so 
action can be taken to avert failure. 

Candidate items must be chosen carefully because the 
analysis can be costly if performed for all potential 
parameters. 
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8.1 Performance Trend Analysis 


8.1.1 Description 

Performance trend analysis, as described in references 8.1 and 8.2, is a parametric assessment of 
hardware and software operations to evaluate their status or to anticipate anomalies or possible 
problems. This assessment not only includes operational performance, such as ballistics of an SRM but 
also assesses hardware performance, such as insulation and inhibitor systems, the motor case, or the 
nozzle system. For example, postflight measurements of insulation indicate the performance of the 
insulation during motor operation. The independent variable in performance trend analysis can be time 
or sequence. Some performance data, for example, that relating to safety, may be recorded and trended 
on a real-time basis. 

As an example, for an SRM, typical operational performance parameters to be trended could be 
peak pressure, total impulse, ignition delay, thrust rise-time characteristics, and propellant structural or 
ballistic properties. Typical hardware performance parameters to be trended could include insulation 
anomalies, structural factor of safety (calculated from as-built drawings), and seal performance (as 
measured, i.e., from leak checks). 

As described in reference 8.2, data sources for performance trend analysis might include new, 
refurbished, and repaired component and subassembly acceptance inspection, checkout, and test data for 
development and verification and production hardware including, but not limited to: 

(1) Alignment data. 

(2) Contamination data. 

(3) Dimensional data. 

(4) Nondestructive test data, e.g., magnetic particle, radiography, penetrant, and ultrasonic 
data. 

(5) Proof test data, e.g., leak check and hydroproof data. 

(6) Functional or performance data, e.g., quantitative and qualitative data. 

8.1.2 Application 

Performance trend analysis is best applied in phase E but may also be applied in phase D. This 
analysis can be used to identify certain parameters that will indicate that a system or system element 
(i.e., subsystem, assembly, subassembly, component and piece-part) is degrading and will potentially 
fail. These parameters can include, but are not limited to, the following: 8 2 

(1) Direct measures of degradation, such as wear, erosion, pitting, and delamination. 

(2) Measures of conditions that might introduce degradation, such as pressure anomalies, 
temperature anomalies, vibration, friction, leakage, and contamination. 

(3) Measures that indicate a shift in performance, such a changes in material properties, 
calibrations, and electrical resistance. 

Attendance to maintenance can help to detect degrading parameters which could lead to failure 
or delay resulting from an exceedance of criteria. 
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8.1.3 Procedures 


The procedures to apply performance trend analysis, adapted from references 8.1 and 8.2, are 
presented below: 

(1) Identify the elements of the system. Assess those hardware or software system elements to 
identify items that could cause critical or costly failures. Each element of the system should 
be considered, i.e., each subsystem, assembly, subassembly, component and piece-part. List 
these system elements as candidates for performance trend analysis. 

(2) From the list, select which items will be analyzed. Concerns (in terms of risk, safety, cost, 
availability, or schedule) and expected benefits should be the basis for setting priorities 
when considering which items to select for performance trend analysis. 

(3) Determine the parameters that characterize the performance of the selected system 
elements. Select parameters that will indicate performance deterioration of the given 
system element in a timely manner for corrective actions to be approved by management 
and implemented. Review the following to identify possible candidate parameters for 
performance trending: 

a. FMEA (sec. 3.4)/critical items list (FMEA/CIL). 

b. Drawings and specifications. 

c. Previous problem reports. 

d. Equipment acceptance data. 

e. Original equipment manufacturer’s data. 

f. Operations manual. 

(4) Establish the criticality of each selected parameter. The parameter criticality should be 
based on the FMEA/CIL or other criteria that have been preapproved by management. The 
criticality of the parameter will indicate the magnitude of the impact if an adverse trend is 
detected and to what level of management that adverse trend is reported. 

(5) Determine if the selected parameters can be quantified with obtainable data. A parameter 
may be quantified with direct measured data (such as temperature, pressure, force, strain, 
acceleration, heat flux, etc.) or by calculation involving two or more direct measurements 
(such as specific impulse for rocket engines or compressor and turbine efficiencies for jet 
engines). If data are not available, establish a system to acquire the data or drop the item 
from trend analysis. 

The availability of the data — the more available the data are, and assuming statistical 
stability, the greater the likelihood of successful trending. Ten to twenty data points for a 
parameter are desirable as a minimum. 

(6) Develop acceptance levels for the parameters. These levels or limits become the basis for 
determining if a parameter is in control or corrective actions are required. First, determine the 
boundaries that define the required range for normal operation. These boundaries should be 
identified for each parameter from a review of vender-supplied data, test or operational data, 
or specifications or requirement documents. Next, determine action limits that fall within 
these boundaries in which corrective action will be initiated if the action limits are exceeded. 
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Care should be taken in choosing the action limits so that (1) variation in normal acceptable 
operation will not cause the action limits to be exceeded (causing unnecessary expenditure 
of resources), and (2) corrective actions can be implemented promptly, once the action limit 
is exceeded but before the boundaries for desired normal operation are exceeded. These 
action limits should be taken from historical data that represent the same distribution for the 
parameter as that in which future measurements will be recorded and tracked. 

(7) Analyze the selected parameters for trends. Various statistical and graphical techniques for 
performing trend analysis can be found in reference 8.3. Use graphical tools to transform 
raw, measured, or calculated data into usable information. The graphical tools can include 
scatter plots (sec. 5.1) and control charts (sec. 5.2). Use statistical tools, such as regression 
analysis (sec. 6.6), to determine the trend line through a given set of performance data. 
Determine how well the trend line fits the data by using techniques such as R 2 or Chi- 
Square measure of fit tests. These tests are described in detail in reference 8.3 and statistical 
textbooks and handbooks. Use the trend line to detect if there is a trend that is approaching 
or has exceeded the action limits determined in step 6. 

(8) Resolve adverse trends. If an adverse trend is detected, determine the cause of the adverse 
trend. Perform correlation analyses (sec. 6.3) to determine what other parameters (factors) 
are contributing to the adverse trend. Once the cause of the adverse trend is identified, 
propose a remedy to correct the problem before the boundaries for desired normal operation 
are exceeded. Implement (management approval may be required) the remedy, then trend 
future performance and assess the effectiveness of the remedy. 

(9) Report the results. To maximize the benefits of the trend analysis effort, the results should 
be documented and distributed to the appropriate levels of management and functional 
organizations to ensure corrective actions are implemented in a timely manner once an 
adverse trend is detected. Typically, these reports should contain the following items 
(adapted from reference 8.2): 

a. System element (from step 1). 

b. Parameter identification (from step 3). 

c. Criticality (from step 4). 

d. Data source (from step 5). 

e. Failure mode as described in the FMEA. 

f. Baseline changes, if applicable. 

g. Indication of excluded data, trends, their direction and disposition (adverse or 
acceptable). 

h. Corrective action used and its effectiveness, if applicable. 

i. Need for additional data, if applicable. 

j. Recommendations, as necessary. 

k. Applicability to other types of trending. 

l. Need for additional correlation analysis, if applicable. 
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8.1.4 Example 


In a machine shop, the service life of saw blades was studied. The objectives of the study were to 
determine the expected life of the blades and develop a methodology to determine when special causes 
were effecting machine performance. Performance trend analysis was performed to address both these 
questions. Blades are replaced when their performance degrades from 10 to 3 cuts per hour. First, 
performance data were collected for 30 blades to statistically establish the expected service life and the 
band for expected normal performance. 

The daily average cuts per hour for each blade of the 30 blades were measured and recorded until 
the 3 cuts-per-hour limit was reached. A linear regression analysis of these data was performed to 
determine the relationship between the cuts per hour and work days. The variation of the 30 blades was 
examined for each day of operation. This analysis revealed that the variation grew linearly with time. A 
band was established from ± 3 standard deviations from the regression line for each day of operation. The 
expected service life range for a given blade was expressed as the time range defined by the regression ± 3 
standard deviation band of the regression intercepted the three cuts-per-hour replacement limit. 

The lower (-3 standard deviation) limit of the band was defined as the action limit to ensure the 
machine is operating properly. The daily average cuts per hour is tracked for a blade in operation. When 
the action limit is exceed, the machine is examined to determine if there is a special cause that is 
reducing the blade service life. 

The expected band for normal operation and expected service life are illustrated on the performance 
trend analysis plot presented in figure 8-1. The performance of a given blade that has just reached the 
end of its service has been tracked on this chart. Note that the action limit is the lower limit of the 
expected normal operation band. 
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8.1.5 Advantages 

(1) Performance trend analysis can be used to detect a degrading parameter prior to a potential 
failure. 

(2) This technique can predict future parameter values or estimate the long-term range of 
values of influential variables. 

(3) The service life of systems or system elements can be predicted. 

8.1.6 Limitations 

(1) Parameter sensors may need to be installed to obtain trending data — this can be costly. 

(2) The operating state, output, or load, about/through which a system/subsystem/component 
fluctuates, often cannot be controlled to achieve consistent trend data. 

(3) The slope and stability of the data approaching/departing the recorded data point are not 
known without using a data buffer. 

(4) Data are not always easily quantifiable, limiting the usefulness of this technique. 

8.1.7 Bibliography 

“ASRM Trend Analysis Requirements Document.” Sverdrup Technology, Inc., Report No. 314-004-91- 
115, December 1991. 

NASA Technical Memorandum 85840, ‘‘The Planning and Control of NASA Programs and Resources.” 

NMI 1103.39, “Role and Responsibilities - Associate Administrator for Safety and Mission Quality 
(SMQ).” 

NMI 8070.3, “Problem Reporting, Corrective Action, and Trend Analysis Requirements.” 

NMI 8621.1, “Mishap Reporting and Investigating.” 

NHB 5300.4 (1A-1), “Reliability Program Requirements for Aeronautical and Space System 
Contractors.” 

NHB 8070.TBD, “Significant Problem Reporting System.” 

Special Study Z001U61, “Marshall Operations Reliability Trend Analysis Standard.” Sept. 16, 1991. 


8.2 Problem Trend Analysis 


8.2.1 Description 

Problem trend analysis, as described in references 8.1 and 8.2, identifies repetitive problems and 
assesses how often given problems occur. Also, problem trend analysis provides a mechanism to track 
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progress of problem resolution. Finally, problem trend analysis evaluates organizational proficiency in 
preventing repetitive problems. Underlying causes can be uncovered when several problems are 
compared. Problem trend analysis is often an early indicator of significant issues in other types of trend 
analysis. 

There are three basic objectives in problem trend analysis: 

(1) Isolate problems to specific causes and examine the frequency of occurrence of these 
causes. Problem trending is often initiated on a system level but finished on a component 
(or lower) level. 

(2) Track problems to determine if occurrence is increasing or decreasing, or if some problems 
are affecting other parameters. 

(3) Determine if baseline changes or corrective actions increase or decrease the frequency of 
problem occurrence. 

Data sources for problem trend analysis may include, but need not be limited to: 

(1) Failure or problem reporting and corrective action systems such as Problem Reporting and 
Corrective Action (PRACA). 8 2 

(2) Discrepancy reports (DR’s). 

(3) Problems identified by the other four types of trend analysis. 

8.2.2 Application 

As described in reference 8.2, problem trend analysis is used to identify recurring problems and 
assesses the progress in resolving these problems and eliminating the recurrence of the problems. This 
analysis is best applied in phase E but may also be applied in phase D. The main interest in this analysis 
is locating where the key problems are occurring and the frequency of occurrence. Graphical techniques 
such as the Pareto analysis (sec. 5.6) are useful in focusing attention and determining where other 
analyses such as performance trend analysis (sec. 8.1) can be beneficial. 

Problem trend analysis provides a historical overview of problems in an easy-to-understand 
graphical format. This overview assists in decision-making relative to design effectiveness, process, or 
procedural changes over time. Problem trend analysis can be the first step in the initiation of corrective 
action to improve system performance. 

Basic criteria (from reference 8.2) for the selection of candidate items include: 

(1) Problem frequency (establish from historical problem report databases). 

(2) Criticality (usually determined from FMEA’s). 

(3) Engineering judgment (by cognizant personnel familiar with both the hardware and 
requirements). 

(4) Unique program or project requirements (these requirements indicate more severe 
consequences than normally associated with a given type of problem). 
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8.2.3 Procedures 


Procedures (adapted from reference 8.2) to perform problem trend analysis are as follows : 

(1) Gather pertinent data. Examine the applicable historical data base(s) and acquire the 
appropriate data. These data bases contain information concerning problem reporting. The 
data bases are usually maintained by the organization responsible for design and 
manufacture of a system element or the operational organization that uses the system. 

Typically, searches are made for rejection rates from acceptance testing, operation 
problems, and configuration nonconformance. These searches should be performed for a 
given time frame. The data bases should be searched for events, operating cycles, hardware 
identification codes (i.e., system, subsystem, assembly, subassembly, component or piece- 
part number), failure mode codes from the FMEA, or key words for given hardware 
failures or failure modes. 

(2) Identify frequency of problems for the system element under consideration. The system 
element may be the subsystem, assembly, subassembly, component or piece-part. 

Determine the number of problems (without distinction of failure mode) associated with the 
system element during given time periods (i.e., days, weeks, months, years, etc.). Next, 
normalize these unrefined frequency data to the number of operations, cycles, missions, or 
elements produced during the given time periods. Construct a bar chart (sec. 5.3) for both 
the unrefined and normalized data. The unrefined data are plotted as a function of 
occurrences versus time, while the normalized data are plotted as a function of occurrence 
rates versus time. 

(3) Identify primary causes of the problems. For each system element under consideration, 
determine the categories of failure modes or causes that induced the problems identified in 
step 2. Careful review of the problem reports should be performed to ensure that 
inconsistent wording of problem reports by different authors does not mask the true value 
of each failure mode or cause. Next, determine the number of occurrences for each failure 
mode or cause. Construct a Pareto chart (sec. 5.6) of the number of occurrences versus 
failure modes or causes and identify areas of concern. From the Pareto chart, identify the 
failure modes or cause of consequence that require further assessment. 

(4) Determine if a trend over time exists for each of the identified failure modes or cause of 
consequence. Normalize the failure mode or cause as the problems were normalized in step 
2 (i.e., normalized by the number of operations, cycles, missions, or elements produced 
during the given time periods). Construct a bar chart (sec. 5.3) for each failure mode or 
cause. These bar charts should present the total and normalized number of occurrences 
versus time. Procedure, process, configuration or design changes and the time of their 
implementation should be noted on these charts. 

Once the bar chart is generated, fit the normalized failure mode or cause occurrences with 
either a linear, exponential, power, logarithmic, or positive parabolic trend line. Determine 
the goodness of fit for each trend line model to the data with such statistical methods as the 
R 2 test. Refer to reference 8.3 or statistical textbooks or handbooks for details in fitting the 
data with trend lines or testing for goodness of fit. 

(5) Report the results. Prepare a summary assessment of the problem trend analysis, including: 

a. System element (from step 2). 

b. Data source, i.e., the historical problem report data base (from step 1). 
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c. Failure modes trended and total number of problem reports assessed. 

d. Criticality (from FMEA) of the failure mode(s) and date of last occurrence. 

e. Baseline procedure, process, configuration or design changes, if applicable. 

f. Chief failure mode or cause of consequence. 

g. Indication of trends, their direction and disposition (adverse or acceptable). 

h. Corrective action used and its effectiveness, if applicable. 

i. Need for additional data, if applicable. 

j. Recommendations, as necessary 

k. Applicability to other types of trending. 

l. Need for additional correlation analysis, if applicable. 

8.2.4 Example 

The monthly rejection rate of wickets exceeded a company’s goal of 5 units per 1,000 units 
produced (0.5 percent) during a 3-mo period last year. A problem trend analysis effort was conducted to 
understand the reason for the increased rejection rate and to formulate a plan to prevent future excessive 
rejection rates. The manufacturing reports for a 1-yr production of wickets were reviewed. The results 
were summarized by month and are presented in figure 8-2 (a). Also, the monthly production and 
rejection rates are shown in figure 8-2(a). 

The cause of each rejection was also identified from the manufacturing problem reports and was 
categorized as being due to human error, inadequate properties of raw materials, production machine 
malfunctions, or other miscellaneous causes. These results are presented for each month in figure 8-2 (b). 

The number of rejections and the rejection rates were plotted on a bar chart and the results are 
presented in figure 8-2(c). The rejection rates were normalized to units produced monthly. As seen on 
this chart, the rejection rate exceeded the company goal of 0.5 percent during August, September, and 
October; therefore, this time period became the focus of the analysis. 

Note from this figure that the normalized rejection rate data, not the absolute number of 
rejections, indicate the time period of concern. 

A Pareto chart (shown in figure 8-2 (d)) was produced for the entire year to establish the 
significance of each cause for rejection. This chart revealed that human error was the most significant 
cause for rejection over the entire year period. However, a Pareto chart generated for the 3-mo period of 
concern, revealed that inadequate material properties was the most significant cause for unit rejection. 
Comparing the two Pareto charts shows that inadequate material properties was a much more significant 
problem during the 3-mo period, and that human error was over the entire year. This chart for the 3-mo 
time period is presented in figure 8-2(e). 

The number of rejections and the rejection rates due to inadequate properties of raw materials 
were plotted on a bar chart and the results are presented in figure 8-2 if). The rejection rates were 
normalized to units produced monthly. As seen on this chart, the increase in the rejection rate due to 
inadequate material properties was the driving factor in exceeding of the maximum rejection goal. 
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Month 

Units Produced 

Units Rejected 

Rejection Rate 
(Units/1,000 Produced) 

Jan. 

5,100 

12 

2.35 

Feb. 

4,600 

21 

4.28 

Mar. 

4,900 

16 

3.26 

Apr. 

2,900 

12 

4.14 

May 

3,150 

13 

4.12 

Jun. 

3,050 

10 

3.27 

Jul. 

3,000 

12 

4.00 

Aug. 

1,700 

14 

10.35 

Sep. 

1,400 

14 

9.65 

Oct. 

1,750 

15 

8.57 

Nov. 

3,100 

9 

2.90 

Dec. 

4,950 

21 

4.24 


(a) History of unit rejections. 


Month 

Causes for Unit Rejection 

Units 

Rejected 

Human 

Error 

Inadequate 

Material 

Properties 

Machine 

Malfunction 

Other 

Jan. 

6 

2 

3 

1 

12 

Feb. 

10 

4 

5 

2 

21 

Mar. 

8 

3 

4 

1 

16 

Apr. 

6 

3 

3 

0 

12 

May 

6 

2 

4 

1 

13 

Jun. 

5 

2 

3 

0 

10 

Jul. 

6 

2 

2 

2 

12 

Aug. 

3 

10 

1 

0 

14 

Sep. 

3 

9 

1 

1 

14 

Oct. 

4 

9 

2 

0 

15 

Nov. 

5 

2 

2 

0 

9 

Dec. 

10 

5 

5 

1 

21 

Total 

72 

53 

35 

9 

169 


(b) History of unit reject by cause. 


Figure 8-2. Problem trend analysis example — Continued 
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(c) Total unit rejection and rejection rate versus time. 



100 

80 

60 

40 

20 

0 


c 

<D 

CJ 

Sh 

<u 

Oh 


Causes of Unit Rejections 
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Figure 8-2. Problem trend analysis example — Continued 
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Rejection Rate, Units / 1,000 Produced 



Properties 


Causes of Unit Rejection 
(for August, September, and October only) 

(e) Pareto chart of causes for period of concern. 
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if) Unit rejection and rejection rate (due to inadequate material properties) versus time. 


Figure 8-2. Problem trend analysis example — Continued. 
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Percent 


Further analysis showed that a statistically significant larger portion of the units rejected for 
material properties came from one lot of materials used during the August to October period. This lot 
met acceptance test criteria, however it was by a narrow margin. To avoid further high rejection rates, 
the specifications for the raw material were tightened as were the corresponding acceptance tests. 

8.2.5 Advantages 

(1) Problem trend analysis can provide an early indicator of significant issues in other types of 
trend analysis. 

(2) This technique examines the frequency of problem occurrence, monitors the progress of 
problem resolution, uncovers recurring problems, and assesses the effectiveness of 
recurrence control. 

8.2.6 Limitations 

Candidate items should be chosen carefully because the analysis can be costly or noninformative 
if performed for all potential problem areas. 

8.2.7 Bibliography 

“ASRM Trend Analysis Requirements Document.” Sverdrup Technology, Inc., Report No. 314-004-91- 
115, December 1991. 

NASA Technical Memorandum 85840, ‘‘The Planning and Control of NASA Programs and Resources.” 

NHB 5300.4 (1A-1). “Reliability Program Requirements for Aeronautical and Space System 
Contractors.” 

NHB 8070.TBD, “Significant Problem Reporting System.” 

NMI 1103.39, “Role and Responsibilities - Associate Administrator for Safety and Mission Quality 
(SMQ).” 

NMI 8070.3, “Problem Reporting, Corrective Action, and Trend Analysis Requirements.” 

NMI 8621.1, “Mishap Reporting and Investigating.” 

Special Study Z001U61, “Marshall Operations Reliability Trend Analysis Standard.” Sept. 16, 1991. 


8.3 Programmatic Trend Analysis 


8.3.1 Description 

Programmatic trend analysis, as described in references 8.1 and 8.2, is concerned with 
organizational or programmatic issues that may impact safety or system success. These issues include 
general program health, schedule issues, overtime or sick time usage, production bottlenecks, accidents 
or equipment damage, supply of critical skills (critical resource scheduling), cost of upkeep versus 
redesign or reprocurement, noncompliances, and cost of replacement versus cost of repair. 
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8.3.2 Application 

Programmatic trend analysis is best applied in phase E. The objective of programmatic trend 
analysis is to provide management a status on programmatic issues or early warning of programmatic 
problems. For example, warning of inappropriate demands on manpower, impending delays, mismatches 
between demand and available expertise, alerting management on areas needing attention (e.g., damage, 
injury or accident frequency), supporting program/project improvement changes, support management in 
monitoring project management performance indicators over time to indicate end-product safety and 
reliability. 

Common candidates for programmatic trend analysis include the following: 

(1) “Manpower strength by speciality, experience, qualification, certification, and grade.” 8 - 2 

(2) “Schedule changes/slippages or overages.” 8 - 2 

(3) Accident or sick time frequency. 

(4) “Overtime usage versus approved policy.” 8 - 2 

(5) Labor problems. 

(6) “Requirement changes, including waivers and deviations.” 8 - 2 

(7) “System nonconformances and problems due to human error.” 8 - 2 

(8) “Rework expenditures.” 8 - 2 

(9) Time/cost considerations for redesign. 

Concerns (in terms of risk, safety, cost, availability, or schedule) and expected benefits should be 
the basis for setting priorities when considering using programmatic trend analysis. 8 2 

Some programmatic trend data will be obtained from other parameters; however, some 
parameters will be unique to programmatic trends. Trending parameters and supporting data to be 
recorded and trended that would have a programmatic impact must be selected. 

8.3.3 Procedures 


As described in reference 8.2, apply the following steps to perform the programmatic trend 
analysis: 

(1) Determine the programmatic parameters to be assessed. Determine which programmatic 
parameters will be trended. Common parameters that are trended are presented in section 
8.3.2. However, the selection of parameters should be determined by the unique needs of 
the organization or program involved. Maintain a list of parameters for which 
programmatic data are to be supplied. 

(2) Acquire and compile data for the selected parameters. Data sources (adapted from reference 
8.2) for programmatic parameters include, but are not be limited to: 
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a. Planned versus actual cost reports (so that number and magnitudes of cost over-runs 
and undermns can be determined). 

b. Planned versus actual schedule charts (so that the number and magnitude of schedule 
delays and accelerations can be determined). 

c. Quality assurance reports (documenting the number of noncompliances). 

d. Development and verification status reports (documenting the success or failure in 
verifying system requirements or specifications). 

e. Inventory control records (documenting the number of times work was delayed due to 
unavailable material). 

f. Facility, equipment, and hardware problem and corrective action reports. 

g. Acceptance records (documenting number of units produced that were accepted or not 
accepted by the customer). 

h. Shipping and receiving logs (including planned versus actual shipping and receiving 
dates). 

i. Work authorization and control documents. 

j. Planned versus actual staff level reports. 

k. Safety, mishap, or incident reports. 

(3) Ensure the validity of the data. Care should be taken to ensure the data analyzed are 
accurate and are an appropriate measure for the programmatic parameter being trended. 

(4) Develop the required analytical techniques and controls (e.g., Pareto charts (sec. 5.6) and 
histograms (sec. 5.7)). Action limits should be establish in which corrective action will be 
initiated if the action limits are exceeded. Action limits can be set to ensure parameters stay 
within the operating and administrative policies and procedures, work standards, and goals 
of the organization. 

(5) Determine the structure for project data collection, maintenance, and reporting. Identify the 
organizations and personnel responsible for collecting, maintaining, assessing, and 
reporting the data. 

(6) Make data available to program management. 

(7) Analyze the data for trends. Use control charts (sec. 5.2) to display the historical trends of 
validated data for the programmatic parameters being measured, along with the realistic 
action limits established. 

(8) Resolve adverse trends. When an adverse trend has been identified, conduct an analysis of 
that trend. Preparing a cause and effect diagram (sec. 7.2) may be useful in identifying the 
root cause of the adverse trend. Once the cause of the adverse trend is identified, propose a 
remedy to correct the problem before the boundaries for desired normal operation are 
exceeded. Implement the remedy (management approval may be required), then trend 
future performance for the programmatic parameter and assess the effectiveness of the 
remedy. 
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(9) Report the results. The reports should be published at intervals that will allow management 
to take prompt action to correct problems before they become unmanageable. The reports 
should contain sufficient details so that management can accurately assess the risk 
associated with an averse trend. Suggested reporting formats for common programmatic 
parameters can be found in reference 8.2. 


8.3.4 Example 

At the start of a new program, candidate parameters were identified for programmatic trend 
analysis. The list was reviewed by both the project team and management, and trending parameters were 
selected. Arrangements were made for data to be collected and assessed for each parameter. Action 
limits were determined from company policies and procedures and program requirements 
documentation. 

The following example illustrates how programmatic trend analysis was applied for a specific 
programmatic parameter — overtime usage. Review of the company policy revealed that the average 
overtime rate for a project with more than the equivalent of 100 full-time workers should not exceed 10 
percent per month. This particular program average staffing level was 125. An action limit of 8 percent 
per month maximum overtime rate was established. If this action limit is approached or exceeded, 
management should be notified and corrective action taken. 

The actual overtime rate, expressed in percentage versus month worked, is presented for 1991 in 
figure 8-3. As seen in this figure, the overtime rate exceeded the action limit in May. Management was 
notified and overtime usage was reviewed. The cause for the increased rate was due to new negotiated 
work to be performed. However, the scheduled completion date for the project had remained fixed. 
Overtime projections revealed that the overtime rate would range from 10 to 13 percent for the 
remainder of the calendar year. 

Work was identified that could be subcontracted. This work was approximately 6 percent of the 
total project. Management agreed to subcontract the work starting in mid-June. Tracking the overtime 
usage rate past the time the corrective action was implemented revealed that the fix of the programmatic 
problem was effective (as shown in fig. 8.3). 

8.3.5 Advantages 

The programmatic trend analysis technique monitors programmatic posture and provides visibility to 
determine the current/projected health of the human support element. 

8.3.6 Limitations 

The data collection process can be extensive because of a potentially large and varied number of 
sources. 

8.3.7 Bibliography 

“ASRM Trend Analysis Requirements Document.” Sverdrup Technology, Inc., Report No. 314-004-91- 
115, December 1991. 

NASA Technical Memorandum 85840, ‘‘The Planning and Control of NASA Programs and Resources.” 
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Figure 8-3. Programmatic trend analysis example. 

NHB 5300.4 (1A-1), “Reliability Program Requirements for Aeronautical and Space System 
Contractors.” 

NHB 8070.TBD, “Significant Problem Reporting System.” 

NMI 1103.39, “Role and Responsibilities - Associate Administrator for Safety and Mission Quality 
(SMQ).” 

NMI 8070.3, “Problem Reporting, Corrective Action, and Trend Analysis Requirements.” 

NMI 8621.1, “Mishap Reporting and Investigating.” 

Special Study Z001U61, “Marshall Operations Reliability Trend Analysis Standard.” Sept. 16, 1991. 

8.4 Supportability Trend Analysis 

8.4.1 Description 

Supportability trend analysis, as described in references 8.1 and 8.2, is performed to evaluate the 
proficiency of an organization at controlling the logistics factors supporting a program. Logistic concerns 
likely to be trended are supplies of spare parts, replaceability, frequency of cannibalization, late deliveries, 
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shortages, maintenance, etc. Typically, data used for supportability trend analysis are not in a form that is 
readily usable. Processing certain data is labor-intensive or may not be feasible due to contractual 
considerations. Sometimes indirect or related parameters may be used to indicate supportability. 

8.4.2 Application 

The supportability trend analysis technique is best applied in phase E. This analysis assesses the 
effectiveness of logistics factors (extracted from reference 8.2) such as the following: 

(1) Maintenance. 

(2) Supply support. 

(3) Facilities management and maintenance. 

(4) Support personnel and training. 

(5) Packaging, handling, storage, and transportation. 

(6) Technical data support. 

(7) Automated data processing hardware/software support. 

(8) Logistics engineering support. 

Supportability trend analysis monitors the current status of the support systems and forecasts the 
future status in order to resolve problems with minimum adverse effect. The current support systems are 
analyzed in order to estimate the future requirements of the systems. Also, support elements that can be 
improved are identified and the effects on the supportability of other program factors are determined. 

Another application of supportability trend analysis is to optimize system availability over 
operating life. This is done by identifying the support elements that can be improved. Also, the effects 
of system reliability and maintainability on supportability are measured, and areas for improvement are 
identified. 

Candidates used to evaluate system reliability/maintainability /availability support characteristics 
include the following: 8 2 

(1) Mean-time-between-failures (MTBF) 

(2) Mean-time-to-repair (MTTR) 

(3) Mean-time-between-repairs (MTBR). 

Concerns (in terms of risk, safety, cost, availability, or schedule) and expected benefits should be 
the basis for setting priorities when considering using supportability trend analysis. 8 - 2 

Supportability trending parameters should be selected that indicate the effectiveness of the 
support elements and the maintainability design factors. Operations and support systems should be 
analyzed, if support degrades, to identify items that could lead to a system failure, schedule delay, or 
cost increase. 
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8.4.3 Procedures 


The procedures (adapted from reference 8.2) to perform supportability trend analysis are as 
follows: 

(1) Assess the overall operation. Identify parameters that could indicate impending system 
failure, cost impacts, and schedule slippages if support functions deteriorate. 

(2) Select parameters to be trended. Determine which parameters (identified in step 1) can best 
be used to evaluate whether support functions are varying at a sufficient rate to require 
management attention. Special consideration should be given to parameters that predict 
system safety or success. 

(3) Determine if quantitative data are available and adequately represent these parameters. 
Supportability parameters may be derived directly from measurements or from calculations 
involving two or more measurements. If measurement data are not available, develop a 
system to measure the data or eliminate the parameter from the list to be trended. 

(4) Establish acceptance limits for the selected parameters. These levels or limits become the 
basis for determining if a parameter is in control or corrective action is required. First, 
determine the acceptance levels and minimum baselines that define the required level of 
support for normal operation. Acceptance limits and minimum support baselines should be 
taken directly from program or project support requirements documentation. These 
boundaries can also be determined from review of operation, maintenance, and logistics 
manuals, and design requirements and specifications documents. 

Next, determine action limits that fall within these boundaries, for which corrective action 
will be initiated if the action limits are exceeded. Care should be taken in choosing the 
action limits so that (1) variation in normal acceptable operation will not cause the action 
limits to be exceeded (causing unnecessary expenditure of resources), and (2) corrective 
actions can be implemented promptly, once the action limit is exceeded, but before the 
boundaries for required support for normal operation are exceeded. 

(5) Gather, measure, or calculate the data to be used to trend the selected supportability 
parameters. Data sources (extracted from reference 8.2) for supportability trend analysis 
may include, but need not be limited to: 

a. Equipment problem reports. 

b. Work authorization documents. 

c. Contractual acceptance records. 

d. Shipping and receiving reports. 

e. Payment records for maintenance. 

f. Transportation records. 

g. Inventory records. 

h. Issues and turn-in records. 

i. Training course attendance records. 
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j. Technical documentation error reporting. 

k. Consumable replenishment records. 

(6) Analyze the selected parameters for trends. Various statistical and graphical techniques for 
performing supportability trend analysis can be found in reference 8.3. Use graphical tools 
to transform raw, measured, or calculated data into usable information. These graphical 
tools can include scatter plots (sec. 5.1), bar charts (sec. 5.3), and control charts (sec. 5.2). 
Use statistical tools, such as regression analysis (sec. 6.6), to determine the trend line 
through a given set of performance data. Determine how well the trend line fits the data by 
using techniques such as R 2 or Chi-Square measure of fit tests. These tests are described in 
detail in reference 8.3 and statistical textbooks and handbooks. Use the trend line to detect 
if there is a trend that is approaching or has exceeded the action limits established in step 4. 

(7) Resolve adverse trends. When an adverse trend has been identified, conduct an analysis for 
that trend. A cause and effect diagram (sec. 7.2) may be useful in identifying the root cause 
of the adverse trend. Once the cause of the adverse trend is identified, propose a remedy to 
correct the problem before the boundaries for required support of normal operation are 
exceeded. Implement the remedy (management approval may be required), then continue to 
trend the supportability parameter and assess the effectiveness of the remedy. 

(8) Report the results. The reports should be published at intervals that will allow management 
to take prompt action to correct support problems before they become unmanageable. The 
reports should contain sufficient details so that management can accurately assess the risk 
to normal operation due to an adverse trend. Suggested reporting formats for common 
supportability parameters can be found in reference 8.2. 


8.4.4 Example 

The following example illustrates supportability trend analysis for inventory control of a specific 
spare part. Review of the project support requirements document revealed that at least eight spare parts 
were always required. To ensure the inventory never reached this level, an action limit of 10 spare parts 
was established. The inventory level for the parts for 11 months in 1988 and 1989 is presented in figure 
8-4. As seen in this figure, the inventory level reached the action level in August 1988. Assessment of 
the cause for the low inventory level revealed that usage of the spare parts did not increase, however, 
more parts received from the vendor were being rejected in acceptance tests. The corrective action was 
to change vendors for the parts. This occurred in September 1988. Tracking the inventory level past the 
time the corrective action was implemented revealed that the fix to the support problem was effective. 

8.4.5 Advantages 

Supportability trend analysis monitors the current health of support systems and forecasts 
support problems to enable resolution with minimum adverse effect. 

8.4.6 Limitations 


Determining the extent of analysis and identifying the appropriate parameter variations that must 
be measured can be difficult. 
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Figure 8-4. Supportability trend analysis example. 
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8.5 Reliability Trend Analysis 


8.5.1 Description 

Reliability trend analysis, as described in reference 8.1, performs a parametric assessment of 
factors affecting system reliability. The objectives of reliability trend analysis are to measure reliability 
degradation or improvement, to predict an out-of-line failure, to verify design certification limits, to 
determine life limits, and to evaluate inspection intervals. Although some parameters will be unique to 
reliability, many parameters pertaining to reliability trending also pertain to performance or problem 
trending. 

Data sources for reliability trend analysis might include new, refurbished, and repaired 
component and subassembly acceptance inspection, checkout, and test data for development and 
verification and production hardware including, but not limited to: 

(1) Alignment data. 

(2) Contamination data. 

(3) Dimensional data. 

(4) Nondestructive test data, e.g., magnetic particle, radiography, penetrant, and ultrasonic 
data. 

(5) Proof test data, e.g., leak check and hydroproof data. 

(6) Functional or performance data, e.g., quantitative and qualitative data. 

8.5.2 Application 

Reliability trend analysis is best applied in phase E but may also be applied in phase D. 
Reliability trending parameters should be selected to indicate changes in the reliability of a system and 
explain their causes. These parameters could also be performance or problem trending parameters or 
strictly reliability parameters. The criteria for selecting parameters should consider criticality, problem 
frequency, engineering judgment, etc. as deemed necessary. Trending parameters should be selected, as 
applicable, for each system, subsystem, or component by: 

(1) For each parameter, reviewing the FMEA/CIL, contract end item specification, limited-life 
items lists, previous problem reports, original equipment manufacturer’s data, equipment 
acceptance data, operations manuals, etc. to determine if it is necessary or beneficial to 
perform reliability trending. 

(2) Determining the product life indicators necessary to determine the health of the system, 
subsystem, or component, e.g., MTBF. 

(3) Determining the failure modes pertinent to the system, subsystem, or component. 

(4) Determining if time/cycle and failure data are available. Typically, at least 10 failures are 
necessary to perform a reliability trend analysis; however, an action limit can be set to 
indicate a “failure” data point. At least half of the time/cycle intervals should have an 
average of at least one “failure” per time period. (For example, if six time intervals of 2 yr 
are chosen, at least three intervals should have at least two failures.) Design/process change 
data should be available. 
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(5) If necessary data are not available (e.g., failure and time/cycle data), consider the addition 
of data sensors, obtaining alternate data, changing the parameter, or using engineering 
judgment for the trend analysis. 

(6) Determining if the parameter concerns reusability /repairability or a one-time failure. 

8.5.3 Procedures 


The only differences between performance and reliability trend analysis are the parameters 
trended. Therefore, the procedures to perform reliability trend analysis are same as presented in section 

8.1.3 for performance trend analysis. 

8.5.4 Example 

This example is a plot of reliability trending where the CSF is plotted for the first 20 flight 
motors. The lines for the mean ± 3 standard deviations are based on the motors up to number 50 and 
give an indication that the later motors have a higher CSF than the first motors. 

Also plotted is the minimum CSF allowable by specification (1.5) that shows that this station is 
well above the requirement. Most stations do not lie this far above the minimum CSF value. 



8.5.5 Advantages 

Reliability trend analysis measures reliability degradation or improvement and enables the 
prediction of possible failures so action can be taken to avert failure. 
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8.5.6 Limitations 


Candidate items must be chosen carefully because the analysis can be costly if performed for all 
potential problem areas. 
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APPENDIX A 
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APPENDIX B 
CASE STUDY: 

TRIALS AND TRIBULATIONS OF USING SYSTEM ENGINEERING TOOLS 
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CASE STUDY: 


TRIALS AND TRIBULATIONS 
OF USING SYSTEM ENGINEERING TECHNIQUES 


The Assignment 


Charlie Smith came in to work early Monday morning. And why not? He was excited! He’d just 
completed a course in System Engineering where they’d shown him all these “techniques” to make his 
job easier and less subjective. He’d known about some of the techniques. But he wished he’d had the 
course about 10 years ago — back when he was just starting as a systems engineer. Well, no matter... 
Today his boss was going to give him a new assignment, and he’d show all of them his newly-found 
proficiency with the toolbox. This should be easy... 

His boss, Mr. Jones came in about 9. It had been hard on Charlie, waiting that long, but he had 
used the extra time to read his mail, do a little filing, return his phone messages, and write a report. 
“Hmmm,” he thought, “maybe I came in a little too early...” 

Aw well, his boss, Jones, had finally made it. Without exchanging pleasantries, Jones gave him a 
package — “It’s your new assignment, but I don’t see how you’ll do it. The boss wants everything 
measurable this time — wants to see how things were decided. Good luck — let me know how it goes.” 
With that, Jones left and Smith tore open the package. “A Hands-on Science Museum display suitable 
for grades K-12, for the Museum’s Chemistry Section.” Since Smith was a designer of aerospace 
hardware, he really wasn’t sure about this one. What was he supposed to do? What were the ground- 
rules? Why get this assignment now, just when he was prepared to use all his training to really produce 
real hardware, not a vague ill-defined thing like this? Smith decided to talk to his boss — this day wasn’t 
getting any better. 

Jones’ secretary let him know that Jones was gone for the day. He went back to his desk and 
found a message from the System Engineering class instructor, Ms. Doe. Puzzled, he called her back, 
but he was so preoccupied with his new task that he started right in talking about it. 

“Can you imagine, I thought I’d get to use those techniques to build something. Guess I’ll have 
to finish this task first though. I think I’ll just pick something and press on. I don’t think the tools apply 
here, you know? It’s not really defined enough and I don’t really think the data even could exist, much 
less that I could get it. I mean, with a problem like this, there really aren’t any 'data’ to look at anyway!” 
Charlie was getting himself kind of worked up. 

Ms. Doe (Jane to her friends) almost laughed when she replied, “Buck (Smith’s boss — no one 
knew his real name) asked me to call you because he thought you might react this way. Now remember 
what we talked about in class. Every problem seems this way at first. The techniques really do add 
value, as long as you think about why you’re using them. Tell you what, why don’t you look through the 
phase A group, think about it, and we can talk it through tomorrow?” 
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Phase A — If at First You Don’t Succeed... 

After calming down a bit, Smith agreed that this was worth a try, but he really didn’t think it 
would work. He hung up, asked his secretary to hold his calls (he liked doing that) and brought out his 
matrix. “Let’s see... trade studies, cost-versus-benefit studies, risk assessment matrix...” No wait, that 
risk assessment thing was a secondary for phase A. He carefully crossed it off his list and continued 
“benchmarking, cause and effect, checklists, and quality function deployment,” all no good, they were 
secondaries. That left brainstorming, Delphi technique, and nominal group technique. Well, that made 
five techniques for him to use. Too bad about quality function deployment — he really liked that one, but 
he’d follow the priorities Ms. Doe had set — after all, she ought to know. 

Smith wanted to be systematic, so he placed the five techniques down in alphabetical order on a 
piece of paper: 

Brainstorming 

Cost-versus-benefit studies 

Delphi technique 

Nominal group technique 

Trade studies. 

He’d start with brainstorming first. Jones was about to ask his secretary to call together his group 
when he started feeling a little silly about asking for their help. After all, he wasn’t sure himself what he 
wanted and didn’t want to look stupid to his employees. “If only this assignment had been better...,” he 
thought. Anyway, (he just wanted to get this done!) he began to brainstorm by himself. 

About an hour later he decided it wasn’t going very well. He had been to museums like the one 
described in his project, and he was a chemical engineer by trade — but so far he just had a page listing of 
potential chemistry-related topics: 

(A) Types of Matter: 

Solid - materials 

Liquid 

Gas 

Plasma 

(B) Types of materials: 

Metal 

Ceramic 

Polymer 

Glass 

(C) Building blocks for materials 

Largest building blocks for materials 
Molecules 

Macromolecules, molecules 
Smaller building blocks for materials 
Atoms 

Electrons 

Neutrons 

Protons 

Subatomic particles 
Quarks, etc. 
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(D) Chemistry 

(E) Designing Materials. 

He didn’t know what kind of exhibit this might make, and brainstorming wasn’t going well. He 
remembered from the class that brainstorming was best performed in a group, but he thought again about 
looking foolish. Well, maybe he’d try the next technique — let’s see, that was cost-versus-benefit studies. 

There was clearly no way to do cost-versus-benefit until he figured out what exhibit he was 
going to build. He remembered from the class that that technique required data, and he didn’t have any. 
He decided not to waste any more time on cost-versus-benefit — he’d be well prepared to argue with Jane 
in the morning. 

The next two methods, Delphi technique and nominal group technique, fell to similar fates. He 
wasn’t really familiar with them. She must have taught them when he was out, checking in with his 
office. That was OK because trade studies, the final technique, was one he liked and had used before. 
Smith began by asking himself what the desirable features of a hands-on museum science exhibit for 
chemistry were. He prepared a listing. 

Features: 

(1) Should be fun as well as educational. 

(2) Should accommodate crowds — not just one person (or if one person, should have pretty 
good throughput). 

(3) Should be sturdy. 

(4) Should have to do with chemistry (he’d almost forgotten that one!). 

(5) Should fit in a space of...? (he’d have to find out). 

(6) Must be ready by...? (he’d have to find out). 

(7) Must cost no more than...? (he’d have to find out). 

(8) Should be interesting to ages 5 through 18. 

(9) Should have minimal consumables (he knew, from experience, that consumables could 
destroy a budget). 

After writing down these features, Smith realized that features 1 and 8 were kind of motherhood 
statements, so he took another cut at figuring what he meant by these: 

(1) Should be fun as well as educational. 

What makes things fun for kids? 

a. It should involve activity, not passive participation (nothing that the kids just watch). 
The younger children might have more activity (crawling, climbing, jumping, running 
etc.) than the older ones. 
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b. It might involve winning or high scores or a competition of some sort. 

c. It might involve their making something — visible accomplishment was usually fun. 

d. It could involve testing of their physical or cognitive powers, but should have a 
black/white answer. 

e. It should not be perceived as childish — must be something an adult would do — don’t 
insult the kids!!! 

f. Moving parts were good — he might want to stay away from solely computerized 
stuff. 

g. High tech was good, maybe having to do with exploration or environmental themes — 
tie it to something they knew something about. 

h. If he wanted to get them to do it again and again, it should be something where they 
might measure improvement, or at least get a different result with a different effort or 
problem. 

Smith sat back, well pleased with this listing. He realized that feature 8 was pretty well covered 
by his current list and decided not to work on it separately. He wanted a little refresher on the trade study 
methodology before he went on, so he pulled out his toolbox. Let’s see... page 2-3 (section 2.1.3) said to 
“ Define the mission objective and requirements for the system under consideration .” All right, he’d done 
that, now what? “ Identify credible alternative candidates for the system under consideration ” — that’s 
what the toolbox said, but how could he do that when he didn’t know what he was building? This 
toolbox thing wasn’t as helpful as he thought. Smith packed up for the day and headed home — tomorrow 
he was going to have a serious talk with Jane. She clearly hadn’t taught this stuff right and anyway, why 
was Buck calling her about his new assignment, and why couldn’t it have been a better one, and... Oh 
well, he’d save all that for tomorrow. 


Phase A — ...Try, Try Again 

It was a bleak, rainy Tuesday morning. Smith’s brooding sleep had been interrupted often by the 
sharp concussions of thunderstorms. He was going to be ready for Jane Doe! He arrived at work and 
pulled together his files. His secretary had managed to get some additional information on the science 
exhibit — the space allowable was approximately 3,000 ft 2 , and his timeframe was approximately 18 mo 
until, it, the museum opened. She had left a note saying that there was still no hard data on his budget 
but it would likely be on the order of $400,000. Well, that was something anyway. He checked his 
calender and found that Jane Doe would be there in about 15 min. He used the time to prepare himself. 

Jane arrived on time, wet, and most infuriating of all, cheerful. “So how did it go yesterday?” 

Smith began, in a controlled but bitter tone: “Poorly. The tools didn’t work very well — there isn’t 
enough data. I went systematically by your procedure — which by the way eliminated some things I 
thought would be very useful — and I don’t think I’ve made any real progress. Another thing...” 
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Jane interrupted him here, with that now-very-annoying laugh, “Slow down, slow down, let’s 
take it one thing at a time. I’ve got all morning, and I think we can make this work. If not, I’ll talk to 
Buck about it. Deal?” 

Smith couldn’t say “no” to that. He figured with just one morning’s effort he’d be able to show 
Jane that this wasn’t going to work — then it would be someone else’s problem. “Deal.” 

They sat down at Smith’s desk. He asked his secretary to hold all calls. (He really liked doing 

that.) 


Smith showed Jane the results of his technique downselection — the alphabetized listing he’d 
taken from the course toolbox. Jane began, “Charlie, you have to remember, the matrix on page 1-7 is 
just a guide . You didn’t have to use all of those techniques that were marked priority 1, or ignore the 
priority 2’s and the unmarked items for that matter. But, since that’s how you started, how did it go?” 
Jane wasn’t wasting any time. 

Smith told her of his difficulties in brainstorming and his concern for calling a team together 
before he had anything planned out. She acknowledged that this was one of the shortcomings of the 
brainstorming technique, and she understood — but didn’t seem to agree with — his reluctance to pull a 
team together. She didn’t want to talk about cost-versus-benefit — she agreed that it wasn’t yet 
appropriate and congratulated Smith on not trying to force-fit the technique to an inappropriate 
application. This was not what Smith had expected. They skipped quickly over the next two techniques. 
She explained they were sort of variations on brainstorming anyway, and got right to his trade study. 
Smith was quite confident of his knowledge of this technique and felt secure that he could show Jane 
once and for all that this project was just inappropriate for his newly mastered skills. 

Jane read his nine features without comment. She then looked at his breakout for feature 1 and 
frowned a bit. Smith didn’t want to lose his opportunity (she was turning out to be pretty hard to comer). 
And he didn’t like that frown one bit. As soon as she looked up he let her have it. “You see, I followed 
the guideline for trade studies — and by the way, I’ve been using them for many years — and couldn’t get 
past the second step. How do I know what specifics to trade when I don’t have any specifics? And how 
can I develop any specifics without data? I just don’t see how this thing is supposed to work!” 

Jane’s response surprised Charlie. “These techniques are only to be used where they can help, 
and you’re the only one who can decide where that is. They don’t replace data. In fact, many of them 
may highlight where data are required, or just how much you don’t know. But, with your specific 
problem, I have a few questions. I would have thought things like safety and access for handicapped 
would be high-priority features. Also, what about education — you’ve analyzed some things that clarify 
fun but what are the specific educational aspects that you’d like to focus on? I think a focus on that 
might help a lot.” 

Charlie knew that throughout the class she’d discussed using the toolbox as a guideline, and that 
it wasn’t necessary to use each technique. He just hadn’t trusted his own knowledge of the toolbox well 
enough to select against the toolbox guidance — cookbooks and go-bys were a lot easier. OK, he’d give 
her that one. That bit about safety and handicapped access — those were good and he added them to his 
listing as features 10 and 11, respectively. As for the educational aspects, that was a great observation. 
Together they began to make up a listing. It didn’t go very well at first, so they called in Dalton from the 
Applied Chemistry department. After about an hour, they had a listing for the top 15 educational areas 
that they wished to focus on: 
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Educational Areas: 


(1) Demonstrate units of mass, length, volume, temperature etc. 

(2) Demonstrate intrinsic (color, viscosity, melting point, hardness, density...) versus extrinsic 
properties (size, shape temperature...). Note intrinsic properties are also known as physical 
properties. 

(3) Demonstrate chemical properties (the tendency of the substance to change, through 
interactions with other substances or singly). 

(4) Demonstrate chemical change (new substance is formed) vs. physical change — include 
exothermic and endothermic changes. 

(5) Demonstrate elements, compounds, mixtures, and solutions. 

(6) Demonstrate the states of matter: solid, liquid, gas, plasma. 

(7) Demonstrate the laws of conservation of mass and energy. 

(8) Provide a feel for Avogadro’s number. 

(9) Demonstrate crystalline nature of many solids. 

(10) Demonstrate the nature of polymer chains. 

(11) Demonstrate the nature of metals and semiconductor materials. 

(12) Demonstrate the principles of catalysis. 

(13) Demonstrate the principles of combustion. 

(14) Demonstrate the special nature of organic chemistry. 

(15) Demonstrate the standard and quantum theory for the atom. 

Smith knew from experience that the next thing to do was to combine these with the features 
listing and see which areas were likely to make exhibits and which might combine, etc. But this sounded 
like a combination of brainstorming and trade studies and checklist all twisted together. He asked Jane if 
that was all right. She explained that there was no problem, as long as what they did was documented 
and reasonably systematic. Charlie felt more like he had while in class — he was starting to get the hang 
of this, again. They decided to brainstorm potential exhibits for each of the 15 specific educational areas, 
and then use the features as a checklist to see if they were satisfied. 

Charlie rewrote the features, renumbering and eliminating items, as appropriate: 

(1) Should accommodate crowds — not just one person (or, if one person, should have pretty 
good throughput). 

(2) Should be sturdy. 
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(3) Must be ready in 18 mo. 

(4) Should be interesting to ages 5 through 18. 

(5) Should have minimal consumables (he knew, from experience, that consumables could 
destroy a budget). 

(6) It should involve activity, not passive participation (nothing that the kids just watched). The 
younger children might have more activity (crawling, climbing, jumping, running etc.) than 
the older ones. 

(7) It might involve winning or high scores or a competition of some sort. 

(8) It might involve their making something — visible accomplishment was usually fun. 

(9) It could involve testing of their physical or cognitive powers but should have a black/white 
answer. 

(10) It should not be perceived as childish — must be something an adult would do — don’t insult 
the kids!!! 

(11) Moving parts were good — he might want to stay away from solely computerized stuff. 

(12) High tech was good, maybe having to do with exploration or environmental themes — tie it 
to something they knew something about. 

(13) If he wanted to get them to do it again and again, it should be something where they might 
measure improvement, or at least get a different result with a different effort or problem. 

(14) Must be safe. 

(15) Should be handicapped-accessible. 

He then rewrote the educational goals, indexing them by lettering them to avoid confusion with 
the numbered features list: 

a. Demonstrate units of mass, length, volume, temperature, etc. 

b. Demonstrate intrinsic (color, viscosity, melting point, hardness, density...) versus extrinsic 
properties (size, shape, temperature...). Note intrinsic properties are also known as physical 
properties. 

c. Demonstrate chemical properties (the tendency of the substance to change, through 
interactions with other substances or singly). 

d. Demonstrate chemical change (new substance is formed) versus physical change — include 
exothermic and endothermic changes. 

e. Demonstrate elements, compounds, mixtures, and solutions. 
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f. Demonstrate the states of matter: solid, liquid, gas, plasma. 

g. Demonstrate the laws of conservation of mass and energy. 

h. Provide a feel for Avogadro’s number. 

i. Demonstrate crystalline nature of many solids. 

j. Demonstrate the nature of polymer chains. 

k. Demonstrate the nature of metals and semiconductor materials. 
1 Demonstrate the principles of catalysis. 

m. Demonstrate the principles of combustion. 

n. Demonstrate the special nature of organic chemistry. 

o. Demonstrate the standard and quantum theory for the atom. 


Phase B — Starting to Have a Ball 

Applied Chemist Dalton suggested that they divide the educational goals among several people. 
Charlie agreed, and decided to work the problem with product development teams. He quickly formed 
several teams and parcelled out the work. That took some explanations! He’d selected Dalton and Jane 
for members of his team, along with design artist Mike Angelo, and a marketing executive who worked 
their company’s precollege outreach efforts, Hewitt Wissard. Their task was to develop exhibits for 
items h, j and k. Jane facilitated the brainstorming session, and by lunchtime they had several concepts 
developed for each of the educational areas. Charlie copied the concept suggestions down from the 
yellow stickies they’d used in brainstorming: 


Provide a feel for Avogadro’s number (item h) 

(1) Build a “ball pit” where the number of balls was some percentage of Avogadro’s number 
and smaller kids could play. (Hewitt had seen something like this at a pizza place and his 
daughter liked it a lot.) 

(2) Have a sugar bed filled with grains of sugar that were some percentage of Avogadro’s 
number. This could also be used for experiments (of some sort) and for microscopy when 
discussing the crystal educational area. Maybe used for eating, too. 

(3) Develop some kind of strength-test thing where kids could compete to get close to 
Avogadro’s Number on a scale or something. (Jane really wasn’t a scientist, but in 
brainstorming, everyone’s input could be important). 
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Demonstrate the nature of polymer chains (item j) 


(1) Have microscopes set up to look at polymer crystals. 

(2) Have a sort of maze set up that was partially amorphous and partially crystalline, like some 
polymers are. Let the kids walk through it. 

Demonstrate the nature of metals and semiconductor materials (item k) 

(1) Have a large blast furnace that the kids could use to heat-treat metals, and then measure the 
resultant properties using an Instron tester. Also have water, oil, and salt quenching baths. 

(2) Set up something where they could provide various amounts of dopant to semiconductor 
crystals, and then measure the resistance etc. 

(3) Have a display showing the crystal structure and how semiconductors and metals work 
(electrically). 

(4) Have polishing wheels set up with microscopes so they could polish specimens and look at 
grain structure and stuff. 

They were far from done, but it was a good start. When Jane asked Charlie if he still wanted her 
to talk to Buck, he was surprised. He’d forgotten the deal during the long morning’s work. “No thanks, I 
admit we’re making pretty good progress. I guess we might even start some phase B work this 
afternoon, huh?” 

“Sure,” replied Jane, “but you’re on your own for a while. I’ve got appointments for the next 
couple of days. I think you’ll do fine, and if you run into problems. I’ll be back Thursday afternoon. 
OK?” 

“Not quite,” said Charlie with his own laugh, “Til see you Friday morning for a team meeting. 

OK?” 


“OK,” laughed Jane, and they all went out to lunch. 

Friday came, and Charlie was impressed with the work they had accomplished. After several 
revisions, they had centered on working on the Avogadro’s number thing, the one with the “ball pit.” 
The decision had come after long discussion, and an aborted attempt to perform a weighted average 
trade study to help quantify the results and the decision-making process. When Jane came in, Charlie 
(and the rest of the group) was eager to find out what they had done wrong in using the trade study 
methodology — although Charlie wasn’t interested in arguing this time. He was kind of looking forward 
to working with Jane again. They showed Jane the brief attempt at a trade study that they had 
formulated: 
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Item 

Criteria Attribute 


Weight Factor 

1 

Avogadro’s number 

(H) 

30 

2 

Fun 

(4, 6, 7,8, 
9,10,11,12) 

30 

3 

Cost 

(5) 

25 

4 

Safe 

(14) 

10 

5 

Variable outcome 

(13) 

5 


Charlie explained that the letter and numbers in parenthesis referred back to the specific feature 
and the educational goal delineations they had previously produced. He was pleased with this, as he 
thought it quite sensible to have tied the criteria attributes to the required features/goals that they had 
agreed upon. Jane agreed that their approach did represent a very logical progression, but Charlie saw 
her half smile again. By now he knew that meant they had made an error, and she had spotted it. 

“Go on,” was all Jane said. 

They let Jane know that they had spent a good deal of time discussing the weighting criteria; it 
represented a group consensus. They then showed her the estimates for each alternative relative to each 
criteria attribute. They had used the TQM techniques of ranking each of the qualitative items as 1, 5, or 9 
to allow separation of results. These rankings, too, were consensus: 


Item 

Criteria Attribute 

Weight 

Factor 

Measure of 
Effectiveness 

Alt 1 

Alt 2 

Alt 3 

1 

Avogadro’s number 

30 

Obvious 

9 

9 

9 

2 

Fun 

30 

See features 

9 

9 

9 

3 

Cost 

25 

ROM estimate 

9 

9 

5 

4 

Safe 

10 

Standard stuff 


9 

9 

5 

Variable outcome 

05 

Obvious 

9 

9 

9 


They hadn’t bothered to calculate the scores or weighted scores. It was obvious that it would come out a 
wash. 


Jane was wearing a large smile now. She said to Charlie, “I think you know where the problem 
was, but I don’t think you recognize the value of what you did! Let’s start with the problem. Tell us 
why you think it didn’t work.” 
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Charlie was again caught off guard — he wished she’d stop doing that — but he answered her 
question. “I don’t think our designs were well formulated enough to be able to evaluate and score. I 
don’t think we did a very good job on defining quantitative, objective criteria attributes. But for this type 
of effort. I’m not sure how to do any better. So how can you use a trade study then, when you’re at this 
stage of a design? Why was it one of the recommended tools?” 

Jane’s eyes glittered with her smile as she began, “You’re right about the criteria attributes. One 
way might have been to simply count the features (an example: How many of the features defining fun 
were incorporated under each alternate?) and quantify how many the alternate would have met. But that 
wouldn’t have solved your fundamental problem. You’re right, the designs really weren’t mature enough 
for this type of study to give a clear selection. To evaluate ‘Safe,’ a PHA would really be required, 
which means you have to have at least a preliminary design. To evaluate ‘Fun’ and ‘Cost,’ the same 
level of maturity is also necessary. But, what I don’t think you’ve realized is by performing this study, 
you were able to identify that. At this stage of design maturity, no concepts were inappropriate. The fact 
that nothing washed out gave you a valuable answer, and let you choose based on ‘gut feel’ — what’s 
often called ‘engineering’ or ‘programmatic judgement.’ Further, you elected to quantify your 
qualitative feel for the alternate by using the 1,5,9 technique. I think you guys did just great! You 
wouldn’t have tried to be specific about why you had selected one idea to pursue if you hadn’t had these 
techniques — you knew intuitively that there weren’t enough data to use appropriate criteria. These 
techniques won’t let you do things that can’t otherwise be accomplished. They’re just an aid. And I think 
you did great. When it wasn’t helping, you tried something else. Which one did you wind up selecting, 
anyway?” 

“Alternate 1- the ball pit,” replied Charlie. “Now I thought we might flowchart the effort 
required for phase B to figure out where we’re going with this. You know — the plan-your-work and then 
work-your-plan kind of stuff.” 

After some long discussions over the blank page that they were trying to use to do a flowchart, 
Jane suggested that a good way to get started might be to flowchart what they had already done. 
Although it seemingly added nothing to their completed tasks, she noted it was often easier to add to 
something, and even easier to edit or change something, than it was to create it in the first place. Starting 
the flowchart with the efforts they had already accomplished would give them a base to add upon, rather 
than the now-beginning-to-be-annoying blank page. They agreed and by the end of the day the group 
had produced a reasonable flowchart (see figure B-l.) Much of the day had been spent on trying to 
define which tools would be used. This time they only used the toolbox as a guide and kept asking, 

“Will this tool help me? Do I need it...?” Their flowchart represented their choices — to their surprise it 
also provided some insights to the design process. 

Many of the phase A decisions were management/programmatic-type decisions that held 
significant consequences for the remainder of the effort. It was also true that most of the data required to 
support credible cost-versus-benefit or risk-versus-benefit trade studies did not exist at this stage. Charlie 
began to hold an even greater appreciation for the toolbox — not so much for the major type decisions, 
but for the potential to reveal the more subtle ramifications of decisions that might otherwise go 
unnoted. He spoke his thoughts to the group, noting that these subtleties were particularly critical at the 
beginning of a project. He received the typical reaction by a group to someone who speaks the 
obvious — they laughed. 


B-13 




Selection Process that Should Have Been Used 

Review Potential tools 
from Handbook for applicability 



^ Don't Use Technique^ 


Toolbox Recommendations 
Technique Familiarity (Personnal) 

Judgement (schedule, criticality, resources etc...) 


the selected 
technique 


Selection Process that Was Used 
to determine Features / Goals 


Additional Step that Should 
Have Been Added 




Downselect Single Concept 




Phase A Flowchart: Note most decisions are 
Programmatic / Management decisions with 
a paucity of data and far-reaching consequences. 


(a) Phase A. 

Figure B-l. Flowchart — Continued 
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Phase B Flowchart: Note from post "flowchart creation" step on, most 
analyses and decisions are technical and discipline-specialist oriented. 


(b) Phase B. 

Figure B-l. Flowchart — Continued. 
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Jane wasn’t laughing though. She noted that there was a change from the fundamental desirable 
attributes of a phase A program manager to those of a phase B manager. Decisions at the phase A level 
really required overall system experience and a capability for accepting available data, determining the 
appropriate level for additional data, and decision selection from the minimum, adequate (often very 
sparse) pool which that represented. Phase B efforts required a much more detail-driven selection 
process which employed the talents and capabilities of discipline specialists — management did well as a 
cheerleader here but might often leave the technical and some of the programmatic optimization 
decisions to the concurrent engineering team working the problem. Phases C and D were similar in 
nature to 
phase A. 

Charlie began to feel a little better about things. He also noted that brainstorming was a 
technique that was seemingly suitable almost everywhere. He and Jane decided to use it to select a 
restaurant for dinner, after the rest of the group had gone home. 

Charlie had gotten home very late Friday night, he was still sleeping when his boss’s secretary, 
Fawn Hunter, called Saturday morning to let him know that first thing Monday morning Buck wanted to 
see what kind of progress they had made and discuss the project. He thanked Fawn for the information 
and got up to do some more work on the flowchart, and to try to start some of the other tools. It was 
looking like it was going to be a long weekend. About an hour later Jane called, volunteering to help. 
Charlie was the one smiling now — the prospects for the weekend were looking up. He liked the phase A 
and B flowcharts and added some of the symbols from the toolbox, just to key in on the difficulties for 
those particular areas. He also added the “should have done” boxes in addition to the “as performed” 
flows, and changed the phase B flowchart accordingly. 

Charlie ran the calculations for the number of marbles in the ball pit: 

Calculation of marble volume: 

Volume of a sphere = (4/3) n r 3 

^marble = 0.5 in 

^marble = 0.125 in 3 

^marble =0.52 in 3 . 

Calculation of ball pit size: 

Assume 20 ftx20 ftx3 ft 


Vballpit = 1,200 ft 3 x 1,728 in/1 ft 3 = 2.07xl0 6 in 3 . 

The maximum packing density for spheres of a single diameter is 74 percent. 
Calculation of number of marbles to “fill ball pit:” 

No. marbles = (0.74) 2.07xl0 6 in 3 /0.52 in 3 = 2.95xl0 6 . 
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Although that was a huge amount of marbles (Charlie started wondering about the feasibility and 
cost of that many marbles), it didn’t begin to approach Avogadro’s number. Charlie was still at a loss for 
how to relate the two, and the diffusion part was still evading him. But now that they had this much of a 
start, he and Jane decided to perform the PHA. Once again it was time for brainstorming. 

They filled up the first page with a list of 1 1 hazards (see figure B-2), first listing all of them, 
then identifying the targets for each of them, then working the severity and probability and risk factor. In 
this way they were able to concentrate on the ideas first, without getting caught up in the specific 
assessment issues or countermeasures. They used the toolbox risk assessment matrix (on page 3-10) that 
had been adapted from MIL-STD-882C. Jane suggested that they only work countermeasures for those 
items with a risk priority code of 1 or 2. There wasn’t any need to improve on those with a 3. Charlie 
was quite surprised to find that the marbles represented such a significant risk. They settled on a design 
change for that item. 

After filling in the countermeasures and the new risk Priority Codes, they were left only with two 
items of code level 2. Charlie didn’t know how to work these and neither did Jane. Jane did mention 
though that they might just be a function of the integrated exhibit area (IEA) — disease transmission in 
children’s museums was a common occurrence and wherever children jumped, someone was liable to 
get jumped on. They decided to go eat a late lunch, pizza, and watch one of these ball pits in action. 

After returning from lunch, Charlie did calculations for larger balls. He had gotten the idea of filling the 
balls with sand or salt to better compare with Avogadro’s number. This also might be useful for partially 
addressing the crystal educational goal. He and Jane worked the new calculations for the larger balls, 
and for a salt filler. 

Calculation of large ball volume: 

volume of a sphere = (4/3) n r 3 

rball = 3.5 in 

r 3 baii = 4.29X10 1 in 3 

Vbaii = 1.8xl0 2 in 3 . 

The maximum packing density for spheres of a single diameter is 74 percent. 

Calculation of number of balls to “fill the ball pit:” 

No. balls = (0.74) 2.07X10 6 in 3 /1.8xl0 2 in 3 = 8.52xl0 3 . 


Volume of a grain of salt: 

volume of a cube = abc 

assume (a) is approximately equal to (b), which is approximately equal to (c) 
assume a = 0.01 in. 
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Esalt grain — (O.Olin) 3 — 1.0x10 ^ in 3 . 

Calculation of the number of grains of salt to “fill ball pit:” 

No. salt grains = 2.07xl0 6 in 3 /1.0xl0- 6 in 3 = 2.07X10 12 . 

The assumption was made that a ball has zero wall thickness and the salt grains will “magically 
stack” in a sperical container. 

Calculation of the number of grains of salt per ball: 

No. grains/ball = (^ball/^salt) = (1 .8x1 0 2 in 3 /1.0xl0“6 in 3 ) 

= 1.8xl0 8 grains/bail 

Calculation of number of grain-filled balls required to house Avogadro’s number of grains: 

No. balls = Av= (1.8x10 s grains/bail) ( x ) = 6.02xl0*- 3 
= 3.34xl0 15 . 

The maximum packing density for spheres of a single diameter is 74 percent. 

Calculation of required ball pit volume to contain 3.34xl0 15 balls: 

Vbaiipit = (Vball) (No. balls) = (1.8X10 2 in 3 ) ( 3.34xl0 13 )/0.74 

= 8.12xl0 17 in 3 = 3,195 mi 3 

Calculation of cube side required to make a cube of volume = 2,364 miles 3 : 

Side = (3,195 miles 3 ) 1 '' 3 = 17.97 mi. 

There. They had made some major progress, and Charlie was beginning to visualize this exhibit. He 
knew they were ready to talk with Buck on Monday. He did want to find out about using the PHA tool 
for programmatic risk evaluation, and he had begun doodling with some small fault trees and was 
impressed by what he could do with them. He had already pretty much decided not to do the PRA 
assessment and.... 


Epilogue...Two (and a Half) for the Show 

Opening day of the museum was a media event. There were speeches and interviews and plenty 
of good words for all. Mr. and Mrs. Smith stayed in the background letting others soak up the limelight. 
They were pleased and proud of what they had done, and excited that their soon-to-be-born child would 
get to visit the museum often. Those lifetime passes for their family turned out to be a great wedding 
gift! Charlie was putting together a short report on the lessons learned during those first few months of 
the project — Jane was going to use it as a case study during her next class on the toolbox. He had left it 
at home for Jane to read, she smiled again as she recalled the listing: 
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(1) The toolbox is just a guide. Use techniques that have value specific to the requirements, not 
simply because they are available or recommended. 

(2) Don’t be afraid to use techniques that you’re unfamiliar with — but get expert help when 
required! Anything can be misused. 

(3) Expect to make mistakes enroute to success. Learn to recognize and correct them. 

(4) Using the techniques does not mitigate the need for facts and data — rather it better defines 
the need (garbage in — garbage out). 

(5) Brainstorming is almost universally useful. 

(6) When she smiles, my wife is always right. 
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GLOSSARY OF TERMS 


Term 

Analysis 
AND Gate 

Backwards Logic 

Barrier 
Basic Event 

Cause 

Common Cause 
Consequence 
Control Limits 

Countermeasure 
Creative Function 


Definition 


An examination of the elements of a system; separation of a whole 
into its component parts. (Reference Section 4.1) 

A logic gate for which an output occurs if all inputs co-exist. All 
inputs are necessary and sufficient to cause the output to occur. 
(Reference Section 3.5) 

The mental process in which an analyst models a system by 
repeatedly asking the question, "What will cause a given failure to 
occur?" Also called top-down logic. (Reference Section 3.0) 

A countermeasure against hazards caused by a flow from an 
energy source to a target. (Reference Section 3.3). 

An initiating fault or failure in a fault tree that is not developed 
further. Also called an initiator or leaf. These events determine 
the resolution limit for a fault tree analysis. 

The event or condition responsible for an action or result. 
(Reference Section 3.10) 

A source of variation that is always present; part of the random 
variation inherent in the process itself. 

Something that follows from an action or condition; the relation of 
a result to its cause. (Reference Section 3.10) 

Limits (also called action limits) set between the mean or nominal 
values of a parameter and specification limits. If a control limit is 
exceeded, corrective actions may need to be implemented before 
the specification limit is exceeded. (Reference Section 5.2) 

An action taken or a feature adopted to reduce the probability 
and/or severity of risk for a hazard. (Reference Sections 3.2 and 
3.4)) 

The means of seeing new ways to perform work by breaking 
through barriers that often stifle thinking. Some techniques that 
are considered creative tools are evolutionary operation (Section 
7.6), brainstorming (Section 7.7), and nominal group technique 
(Section 7.10). (Reference Table 1-1) 
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GLOSSARY OF TERMS 


Term 


Definition 


Critical Items List 
(CIL) 

A FMEA-derived list (published as FMEA/CIL) containing system 
items that have a criticality of 1 or 2, and items that are criticality 
1R or 2R and fail redundancy screens. (Reference Section 3.4) 

Criticality 

In reference to a parameter, criticality is the level of importance the 
parameter has to the operation of the system. (Reference Section 
3.4) 

Customer 

The internal or external person or organization that is the user of a 
product being produced or service being rendered. The immediate 
customer is the user of the product or service in the next step of the 
process. 

Cut Set 

Any group of fault tree initiators which, if ah occur, will cause the 
TOP event to occur. (Reference Section 3.6) 

Data Analysis 
Function 

The means of analyzing a process by using a data display. Some 
techniques that are considered data analysis tools are checklists 
(Section 7.8), control charts (Section 5.2), and force field analysis 
(Section 7.11). (Reference Table 1-1) 

Decision Making 
Function 

After analyzing all available data, a decision is made on how to 
optimize the subject process. Some techniques that are considered 
decision making tools are benchmarking (Section 7.1), nominal 
group technique (Section 7.10), and force field analysis (Section 
7.11). (Reference Table 1-1) 

Degrees of Freedom 

The number of independent unknowns in the total estimate of a 
factorial effect or a residual. (Reference Section 6.2) 

Facilitator 

A person trained in group dynamics and problem-solving 
structures who assumes the responsibility for ensuring a full 
exchange of information between team members. (Reference 
Section 7.2) 

Factor 

A parameter or variable that affects product/process performance. 
(Reference Section 6.2) 

Fail Safe 

Proper function is impaired or lost but no further threat of harm 
occurs. (Reference Section 3.4) 

Failure 

A fault owing to breakage, wear out, compromised structural 
integrity, etc. (Reference Section 3.4) 
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GLOSSARY OF TERMS 


Term 

Failure Domain 

Failure Mode 

Failure Propagation 
Path 

Fault 

Forward Logic 
Graphical Function 

Hazard 

Intermediate Event 
Mean 

Mean Square 
Deviation (MSD) 

Mishap 

Modeling Function 


Definition 


In analysis work, failure domain refers to an analysis that seeks 
the probability of a system not operating correctly. (Reference 
Section 3.8) 

The manner in which a failure occurs, i.e. the manner in which it 
malfunctions. (Reference Section 3.4) 

The sequence of events that leads to an undesirable event or loss. 
Also called an accident sequence. 

Inability to function in a desired manner, or operation in an 
undesired manner, regardless of cause. (Reference Section 3.6) 

The mental process in which an analyst models a system by 
repeatedly asking the question, "What happens when a given 
failure occurs?" Also called bottom-up logic. (Reference Section 
3.0) 

The means of analyzing the data of a process by applying graphs 
and/or charts. Some of the techniques that are considered 
graphical tools are cause and effect diagram (Section 7.2), control 
charts (Section 5.2), and quality function deployment (Section 
7.12). (Reference Table 1-1) 

An activity or condition which poses a threat of loss or harm; a 
condition requisite to a mishap. (Reference Section 3.2) 

An event that describes a system condition produced by preceding 
event and contributing to later events. 

The term used to describe a sample population average. 

(Reference Section 6.1) 

A measure of variability around the mean or target value. 


An undesired loss event. (Reference Section 8.3) 

The means of analyzing and modeling a process against standards 
and/or other processes. Some of the techniques that are 
considered modeling tools are benchmarking (Section 7.1), quality 
function deployment (Section 7.12), and work flow analysis 
(Section 7.16). (Reference Table 1-1) 
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GLOSSARY OF TERMS 


Term 

OR Gate 

Parameter 
Path Set 

Population 

Preliminary 

Prevention Function 

Probability 

Problem 

Identification 

Function 

Process 

Project Phase A 


Definition 


A logic gate in which an output occurs if one or more inputs exist. 
Any single input is necessary and sufficient to cause the output to 
occur. (Reference Section 3.5) 

The term applied to population or sample characteristics such as 
the mean and standard deviation. (Reference Section 5.2) 

A group of fault tree initiators which, if none of them occurs, will 
guarantee that the TOP event cannot occur. (Reference Section 

3.6) 

The universe of data under investigation from which a sample will 
betaken. (Reference Section 6.1) 

Coming before and usually forming a necessary prelude to 
something. As in a preliminary hazard analysis, the analysis can 
be performed in the design or pre-operation phase, or it can be the 
first analysis performed for a mature system. (Reference Section 

3.2) 

The means of analyzing data to be able to recognize potential 
problems and prevent the process from heading in an adverse 
direction. Some of the techniques that are considered preventive 
tools are control charts (Section 5.2), Pareto analysis (Section 

5.6) , and design of experiments (Section 7.5). (Reference Table 
1 - 1 ) 

The liklihood an event will occur within a defined time interval. 
(Reference Section 3.14) 

The means of identifying potential problems from a data display as 
a result of an analysis of the process. Some techniques that are 
considered problem identification tools are control charts (Section 

5.2) , brainstorming (Section 7.7), and quality function 
deployment (Section 7.12). (Reference Table 1-1) 

A series of events progressively moving forward over time to 
produce products or services for a customer. (Reference Section 
7.1) 

The conceptual trade studies phase of a project. Quantitative 
and/or qualitative comparison of candidate concepts against key 
evaluation criteria are performed to determine the best alternative. 
(Reference Section 1.3) 
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GLOSSARY OF TERMS 


Term 

Project Phase B 

Project Phase C 

Project Phase D 

Project Phase E 

Qualitative 

Quantitative 

Range 

Raw Data 
Reliability 
Risk 
Sample 

Severity 
Special Cause 


Definition 


The concept definition phase of a project. The system mission and 
design requirements are established and design feasibility studies 
and design trade studies are performed during this phase. 
(Reference Section 1.3) 

The design and development phase of a project. System 
development is initiated and specifications are established during 
this phase. (Reference Section 1.3) 

The fabrication integration, test, and evaluation phase of a project. 
The system is manufactured and requirements verified during this 
phase. (Reference Section 1.3) 

The operations phase of a project. The system is deployed and 
system performance is validated during this phase. (Reference 
Section 1.3) 

Data that are not numerical in nature. (Reference Section 2.1) 

Data that are numerical in nature or can be described numerically. 
(Reference Section 2.1) 

A measure of the variation in a set of data. It is calculated by 
subtracting the lowest value in the data set from the highest value 
in that same set. (Reference Section 5.2) 

Data as measured or as taken directly from instruments or sensors. 
(Reference Section 8.4) 

The probability of successful operation of a system over a defined 
time interval. (Reference Section 3.3) 

For a given hazard, risk is the long-term rate of loss; the product 
of loss severity and loss probability. (Reference Section 3.1) 

One or more individual events or measurements selected from the 
output of a process for purposes of identifying characteristics and 
performance of the whole. (Reference Section 6.1) 

The degree of the consequence of a potential loss for a hazard. 
(Reference Section 3.1) 

A source of variation that is intermittent, unpredictable, unstable; 
sometimes called an assignment cause. It is signalled by a point 
beyond the control limits. (Reference Section 8.1) 
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GLOSSARY OF TERMS 


Term 

Standard Deviation 

Subassembly 
Success Domain 

System 

System Element 
Target 

Threat 
TOP Event 

Trends 

Upper Control Limit 
Range 

Variation 
Weighting Factor 


Definition 


A measure of variability used in common statistical tests. The 
square root of the variance. (Reference Section 6.1) 

A composite of components. (Reference Section 3.4) 

In analysis work, success domain refers to an analysis that seeks 
the probability of a system operating correctly. (Reference Section 
3.8) 

A composite of subsystems whose functions are integrated to 
achieve a mission (includes materials, tools, personnel, facilities, 
software, and equipment). 

A constituent of a system that may be a subsystem assembly, 
component, or piece-part. 

An object having worth that is threatened by a hazard. The object 
may be personnel, equipment, downtime, product, data, 
environment, etc. (Reference Section 3.1) 

A potential for loss. A hazard. (Reference Section 3.1) 

The conceivable, undesired event to which failure paths of lower 
level events lead. (Reference Section 3.6) 

The patterns in a run chart or control chart that feature the 
continued rise or fall of a series of points. Like runs, attention 
should be paid to such patterns when they exceed a predetermined 
number (statistically based). (Reference Section 8.0) 

The upper control limit for the moving range chart for a set of data. 
(Reference Section 7.14) 


The inevitable difference among individual outputs of a process. 
The sources of variation can be grouped into two major classes: 
Common Causes and Special Causes. (Reference Section 6.2) 

A method of rating the relative importance of a concern or selection 
criterion as related to comparable concerns or selected criteria. 
(Reference Sections 2.1 and 7.12) 
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HAZARDS CHECKLIST 


Electrical 

Shock 

Burns 

Overheating 

Ignition of Combustibles 

Inadvertent Activation 

Power Outage 

Distribution Backfeed 

Unsafe Failure to Operate 

Explosion/Electrical (Electrostatic) 

Explosion/Electrical (Arc) 


Mechanical 

Sharp Edges/Points 

Rotating Equipment 

Reciprocating Equipment 

Pinch Points 

Lifting Weights 

Stability/Topping Potential 

Ejected Parts/Fragments 

Crushing Surfaces 


Acceleration/Deceleration/ Gravity 

Inadvertent Motion 

Loose Object Translation 

Impacts 

Falling Objects 

Fragments/Missiles 

Sloshing Liquids 

Slip/Trip 

Falls 


Temperature Extremes 

Heat Source/Sink 

Hot/Cold Surface Bums 

Pressure Evaluation 

Confined Gas/Liquid 

_ Elevated Flammability 

Elevated Volatility 

Elevated Reactivity 

Freezing 

Humidity/Moisture 

Reduced Reliability 

Altered Structural Properties 

(e.g., Embrittlement) 


Pneumatic/Hydraulic Pressure 

Overpressurization 

Pipe/Vessel/Duct Rupture 

Implosion 

Mislocated Relief Device 

Dynamic Pressure Loading 

Relief Pressure Improperly Set 

Backflow 

Crossflow 

Hydraulic Ram 

Inadvertent Release 

Miscalibrated Relief Device 

Blown Objects 

Pipe/Hose Whip 

Blast 


Radiation (Ionizing) 

Alpha 

Beta 

Neutron 

Gamma 

X-Ray 


Radiation (Non-Ionizing) 

Laser 

Infrared 

Microwave 

Ultraviolet 


Notes: 

1 . Neither this nor any other hazards checklist should be considered complete. This list should be enlarged as 
experience dictates. This list contains intentional redundant entries. 

2. This checklist was extracted from "Preliminary Hazard Analysis (Lecture Presentation)", R.R. Mohr, Sverdrup 
Technology, Inc., June 1993 (Fourth Edition). 
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HAZARDS CHECKLIST 


Fire/Flammability - Presence of: Leaks/Spills (Material Conditions) 


Fuel 

Liquid/Crvogens 

Ignition Source 

Gases/Vapors 

Oxidizer 

Dusts - Irritating 

Propellant 

Radiation Sources 
Flammable 

> (Initiators) 

Toxic 

Reactive 

Heat 

Corrosive 

Friction 

Slipperv 

Impact/Shock 

Odorous 

Vibration 

Pathogenic 

Electrostatic Discharge 

Asphvxiating 

Chemical Contamination 

Flooding 

Lightning 

Run Off 

Welding (Stray Current/Sparks) 

Vapor Propagation 


Explosives (Effects) 

Mass Fire 

Blast Overpressure 

Thrown Fragments 

Seismic Ground Wave 

Meteorological Reinforcement 


Chemical/Water Contamination 

System-Cross Connection 

Leaks/Spills 

Vessel/Pipe/Conduit Rupture 

Backflow/Siphon Effect 


Explosives (Sensitizes) 

Heat/Cold 

Vibration 

Impact/Shock 

Low Humidity 

Chemical Contamination 


Explosives (Conditions) 

Explosive Propellant Present 

Explosive Gas Present 

Explosive Liquid Present 

Explosive Vapor Present 

Explosive Dust Present 


Physiological (See Ergonomic) 

Temperature Extremes 

Nuisance Dusts/Odors 

Baropressure Extremes 

Fatigue 

Lifted Weights 

Noise 

Vibration (Raynaud's Syndrome) 

Mutagens 

Asphyxiants 

Allergens 

Pathogens 

Radiation (See Radiation) 

Cryogens 

Carcinogens 

Teratogens 

Toxins 

Irritants 


Notes: 

1 . Neither this nor any other hazards checklist should be considered complete. This list should be enlarged as 
experience dictates. This list contains intentional redundant entries. 

2. This checklist was extracted from "Preliminary Hazard Analysis (Lecture Presentation)", R.R. Mohr, Sverdrup 
Technology, Inc., June 1993 (Fourth Edition). 
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HAZARDS CHECKLIST 


Human Factors (See Ergonomic) 

Operator Error 

Inadvertent Operation 

Failure to Operate 

Operation Early/Late 

Operation Out of Sequence 

Right Operation/Wrong Control 

Operated Too Long 

Operate Too Briefly 


Unannunciated Utility Outages 

Electricity 

Steam 

Heating/Cooling 

Ventilation 

Air Conditioning 

Compressed Air/Gas 

Lubrication Drains/Slumps 

Fuel 

Exhaust 


Ergonomic (See Human Factors) 

Fatigue 

Inaccessibility 

Nonexistent/Inadequate "Kill" 

Switches 
Glare 

Inadequate Control/Readout 

Differentiation 

Inappropriate Control/Readout 

Location 

Faulty/Inadequate 

Control/Readout Labeling 

Faulty Work Station Design 

Inadequate/Improper Illumination 


Control Systems 

Power Outage 

Interferences (EMEESI) 

Moisture 

Sneak Circuit 

Sneak Software 

Lightning Strike 

Grounding Failure 

Inadvertent Activation 


Common Causes 

Utility Outages 

Moisture/Humidity 

Temperature Extremes 

Seismic Disturbance/Impact 

Vibration 

Flooding 

Dust/Dirt 

Faulty Calibration 

Fire 

Single-Operator Coupling 

Location 

Radiation 

Wear-Out 

Maintenance Error 

Vermin/V armints/Mud Daubers 


Contingencies (Emergency Responses by 
System/Operators to "Unusual" Events): 


"Hard" Shutdowns/Failures 

Freezing 

Fire 

Windstorm 
Hailstorm 
Utility Outrages 
Flooding 
Earthquake 
Snow/Ice Load 


Notes: 

1 . Neither this nor any other hazards checklist should be considered complete. This list should be enlarged as 
experience dictates. This list contains intentional redundant entries. 

2. This checklist was extracted from "Preliminary Hazard Analysis (Lecture Presentation)", R.R. Mohr, Sverdrup 
Technology, Inc., June 1993 (Fourth Edition). 
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HAZARDS CHECKLIST 


Mission Phasing 

Transport 

Delivery 

Installation 

Calibration 

Checkout 

Shake Down 

Activation 

Standard Start 

Emergency Start 

Normal Operation 

Load Change 

Coupling/Uncoupling 

Stressed Operation 

Standard Shutdown 

Shutdown Emergency 

Diagnosis/Trouble Shooting 

Maintenance 


Notes: 

1 . Neither this nor any other hazards checklist should be considered complete. This list should be enlarged as 
experience dictates. This list contains intentional redundant entries. 

2. This checklist was extracted from "Preliminary Hazard Analysis (Lecture Presentation)", R.R. Mohr, Sverdrup 
Technology, Inc., June 1993 (Fourth Edition). 
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APPENDIX E 

EXAMPLE PRELIMINARY HAZARD ANALYSIS WORKSHEET 
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Example Preliminary Hazard Analysis Worksheet 



*This worksheet was extracted from “Preliminary Hazard Analysis (Lecture Presentation),” R.R. Mohr, Sverdrup Technology, Inc., June 1993. 



















APPENDIX F 

EXAMPLE FAILURE MODES AND EFFECTS ANALYSIS WORKSHEET 
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Example Failure Modes And Effects Analysis Worksheet* 



*This worksheet was extracted from “Failure Modes and Effects Analysis (Lecture Presentation),” R.R. Mohr, Sverdrup Technology, Inc., July 1993. 


